[Max]

I've got a list of about 50,000 lines of items, some of which are probably (read: definitely) duplicates. I'd like to de-dupe them either with PHP or possibly a bash script...anyone have any snippets to do this efficiently?

(Yes, I could write a small script to do this but my guess is that some has a nifty snippet already tucked away somewhere, and it's probably smarter and faster than what I'd churn out.)

[phpMan2010]

cut | sort | uniq



If you can post a couple of lines, I can put more details

[Max]

Here's a sample:

100ideas.com
1dog.net
1dollargreetings.com
1dollarnoni.com
bevirusproof.org
beyond24.com
beyond30k.com
beyondigital.com
beyondspecials.com
custhelp.com
customadultmalls.com
customer-contact.net
customer-svc.com
customerblast.com
customerparadigm.com
customizejacketsandtees4you.com
customoffers.com
customoffersmail.com
cut-to-the-chase.com
cuteandcuddly.com
emaildealz.net
emaildelivery.org
emaildelvery.org
emaildownunder.com
emailfactory.com
emailgaul.com
emailgids.net

[Keith]

This may use upwards of a meg of memory or so, but it sounds like a one-and-done thing (possibly on your local machine?)  so no worries.

Code:  select

Highlight Mode: (PHP)
$lines = file( './domains.txt', FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES );
$lines = array_unique( $lines );
$lines = implode( PHP_EOL, $lines );
file_put_contents( './domains.unique.txt', $lines );
 

...and by replacing array_unique() with array_flip()/array_keys() you can cut the processing time in half-ish:

Code:  select

Highlight Mode: (PHP)
$lines = file( './domains.txt', FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES );
$lines = array_flip( $lines );
$lines = array_keys( $lines );
$lines = implode( PHP_EOL, $lines );
file_put_contents( './domains.unique.txt', $lines );
 

[phpMan2010]

On the command line:

Code:  select

Highlight Mode: (Bash)
sort listofdomains | uniq > uniqed
 

With PHP:

Code:  select

Highlight Mode: (PHP)
<?php
$list=explode(PHP_EOL,`sort listofdomains | uniq`);
var_dump($list);
 

[Max]

Code:  select

Highlight Mode: (Bash)
sort listofdomains | uniq > uniqed
 

Awesome- exactly what I was looking for, a slick one-liner. Thank you!

[Max]

phpMan's bash script is perfect for what I need, but I'm also going to use one of your snippets for some other de-duping I need to do on a regular basis (via cron). My server restricts non-root users from running eval(), passthru(), or exec() so I can't use a bash script from within a PHP script...but I can use what you provided. Thank you.

This may use upwards of a meg of memory or so, but it sounds like a one-and-done thing (possibly on your local machine?)  so no worries.

Code:  select

Highlight Mode: (PHP)
$lines = file( './domains.txt', FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES );
$lines = array_unique( $lines );
$lines = implode( PHP_EOL, $lines );
file_put_contents( './domains.unique.txt', $lines );
 

...and by replacing array_unique() with array_flip()/array_keys() you can cut the processing time in half-ish:

Code:  select

Highlight Mode: (PHP)
$lines = file( './domains.txt', FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES );
$lines = array_flip( $lines );
$lines = array_keys( $lines );
$lines = implode( PHP_EOL, $lines );
file_put_contents( './domains.unique.txt', $lines );