Anime CRC32 Checksum in Linux v2.0

Posted by Taoufix on Jan 2nd, 2009 in Computers

A little work on my previous Anime CRC32 checksum program resulted in this:

Anime CRC32 Check

A standalone version (no extra python modules needed) to check your favorite Amine CRC32 sums :)

#!/usr/bin/python

# Version 0.2 2009.01.25

# Copyright (c) 2009, Taoufik El Aoumari

# Released under the GPL license http://www.gnu.org/licenses/gpl-3.0.txt

import sys, re, zlib, os

c_null  = "\x1b[00;00m"

c_red   = "\x1b[31;01m"

c_green = "\x1b[32;01m"

p_reset = "\x08"*8

def crc32_checksum(filename):

    crc = 0

    file = open(filename, "rb")

    buff_size = 65536

    size = os.path.getsize(filename)

    done = 0

    try:

        while True:

            data = file.read(buff_size)

            done += buff_size

            sys.stdout.write("%7d"%(done*100/size) + "%" + p_reset)

            if not data:

                break

            crc = zlib.crc32(data, crc)

    except KeyboardInterrupt:

        sys.stdout.write(p_reset)

        file.close()

        sys.exit(1)

    sys.stdout.write(p_reset)

    file.close()

    if crc < 0:

        crc &= 2**32-1

    return "%.8X" %(crc)

for file in sys.argv[1:]:

    try:

        crc = crc32_checksum(file)

        dest_sum = re.split("([a-fA-F0-9]{8})", file)[-2]

        if crc == dest_sum.upper():

            c_in = c_green

        else:

            c_in = c_red

        sfile = file.split(dest_sum)

        print("%s%s%s   %s%s%s%s%s" % (c_in, crc, c_null, sfile[0], c_in, dest_sum, c_null, sfile[1]))

    except(IndexError, ValueError):

        print(crc, "   ", file)

    except (IOError, e):

        print(e)

        continue

Note: This code was only tested under Python 2.5 and 2.6. I haven’t tested it (yet) with version 3.0.
Edit 2009.01.05: Corrected the regular expression thanks to Redditor Ademan suggestion.
Edit 2009.01.15: Added percentage progress per file and clean exit on keyboard interruption (Ctrl+C).
Edit 2009.01.25: Split using the checksum itself and ignore case in checksums. thanks to Arfy suggestions.
Edit 2009.05.13: Added GPL license.

35 Responses for "Anime CRC32 Checksum in Linux v2.0"

Jan 5th

skoruppa

Nice but if you have a directory in folder, you script return this
“[Errno 21] Is a directory” its not a big problem but just looks ugly ;)
Jan 5th

skoruppa

And useful to have any exceptions. Like subtitles (ass, srt)

58D5C278 [gg-BSS]_Gundam_00_S2_-_01_[44C8CD36].ass

;)
Jan 25th

Arfy

Folks, it’s easy enough to simply specify what you DO want to check.

arfy@geass:/mymedia/anime/Nodame Cantible$ animecheck *.mkv

Taoufix: THANK YOU for this very useful code! I have not found anything else out there that does this! And nice thinking using zlib’s CRC routine btw.

Suggestion: I think would be useful to perhaps integrate this filename checking as a module or option for cfv (http://cfv.sourceforge.net/) which incidentally is also written in python.
Jan 25th

Arfy

OK feel free to delete my THREE previous posts, cause here I offer solutions.

Line 38, to allow for not only [CRC] but also {CRC} (CRC) _CRC_ or spaces:

dest_sum = re.split(”([\[\{$_ ][A-Fa-f0-9]{8}[\]\}$_ ])”, file)[-2][1:9]

Line 38, to not check for delimiters at all (assumes sane filenames otherwise) note the range 1:9 has been changed to 0:8 .. I don’t know python I’m just guessing right :o)

dest_sum = re.split(”([A-Fa-f0-9]{8})”, file)[-2][0:8]

Line 39, to allow for lower-case a-f in crc as included in the regexes above:

if crc == str.upper(dest_sum):

Hopefully others will find this edit useful.
Jan 25th

Taoufix

@Arfy: Thanks a lot for your comments. I think it’s better to split the filename using the checksum itself, this way we avoid testing for every separator out there ;). Anyway, I updated the script to solve the issues you had.
Jan 27th

Arfy

It is not probably as great an idea as it seems to get rid of the delimiters altogether, especially if you allow for lowercase. Filenames can be split at the wrong points if they contain any 8 chars in a row that match [a-fA-F0-9] For example, “decafbad” and part of “deadbeefcafe” etc will match erroneously and throw off the checking. Of course theses cases will generally be infrequent, but I choose to use the delimiters {} [] () __ to avoid unexpected results. It really comes down to the dataset you wind up working with. I don’t think I’ve seen lowercase checksums in my anime.
Feb 15th

Xiong Chiamiov

Ooh, nice! This is super-useful to me.

I’m thinking you might want to apply a filter (filter, is that the right term for Python?) on the file list, to remove things that don’t have a dot in the filename (I suppose you could make an exception for dots at the beginning, but do you really want to check hidden files?). It should be fairly simple, but I can’t do it off the top of my head.

BTW, 2nd sentence s/Amine/anime/. Also, your captcha doesn’t work w/o javascript, which you might want to mention somewhere.
Mar 4th

Fahad Yousuf

:D Awesomeness. I was half way through writing my own script when I came accross this. Good work!!
Jun 11th

Batousay

First of all thanks for your awsome program.

Today I’ve passed form Arch Linux to Ubuntu and I get this error when I try to execute animecheck:
[batousay@Batu ~]$ ./animecheck \[NTF\]Naruto\ Shippuden\ 111\ \[\].mp4
File “./animecheck”, line 34
if crc < 0:
^
SyntaxError: invalid syntax
[batousay@Batu ~]$ python –version
Python 2.6.2

Thanks for your atention.
Jun 11th

Batousay

It seemed to be HTML codification…

if it can help someone like me…

< == <
& == &

[bye]
Jun 11th

Taoufix

@Batousay: Thanks for your feed back, it fixed now.
Jul 6th

DracoP

I am a nOOb, how do you create this script?
Jul 6th

Xiong Chiamiov

@DracoP

Copy the script into a text file, save (preferably as something.py), then run it. For instance, if I had saved it as foo.py in ~/bin, and I was in a directory with a bunch of Matroska files I wanted to check, I’d run

python ~/bin/foo.py *.mkv

Alternatively, if you give the script executable permissions (chmod +x foo.py), then you can just run it like so:

~/bin/foo.py *.mkv

and, if ~/bin is in your PATH,

foo.py *.mkv

HTH
Jul 6th

DracoP

Is there a way to write the script so that it will be an option on the second-click drop down menu? I know that it is possible as sometime ago, I had a script that coverted wma to mp3 the same way.
Jul 7th

Xiong Chiamiov

That all depends on what filebrowser you’re using. Linux is not one cohesive entity.
Jul 7th

DracoP

Using ubuntu, so nautilus I guess.
Jul 7th

Taoufix

@DracoP: “Is there a way to write the script so that it will be an option on the second-click drop down menu”

I don’t know about that but you can add an “Open In Terminal” entry.

Open a terminal and type in:
[shell]mkdir ~/bin/
gedit ~/bin/animecheck
[/shell]
Copy/Past the script in there, save, exit, go back to the terminal and make the script executable:
[shell]chmod 755 ~/bin/animecheck[/shell]
Like Xiong suggested, add the ~/bin/ folder to your PATH:
[shell]echo ‘export PATH=${HOME}/bin:${PATH}’ >> ~/.bashrc[/shell]
Now install nautilus-open-terminal:
[shell]sudo aptitude install nautilus-open-terminal[/shell]
Log out and log back in. Now you can browse to your Anime folder, right-click, click on “Open In Terminal” then type in:
[shell]animecheck *[/shell]
Jul 8th

DracoP

Works like a charm.
Jul 14th

RiCON

Hi, Taoufix.

I’m using a bit of your code (most of the crc32_checksum function, more precisely) for a script of mine. This is the first time I wrote a script using other people’s work. Seeing as you licensed it to GPLv3, what do I have to do in case I release my script publicly? Do I have to license it as GPLv3 as well, mentioning I’m using part of your code or can I use another license?
Jul 14th

Xiong Chiamiov

You don’t have to attribute anything to him, but you do have to release the derivative version under the GPL. Take into account that IANAL, and it’s been a while since I’ve read the GPL.

You’ll find lots of questions about this kind of thing on Stack Overflow, and lots of informative comments on Jacob Kaplan-Moss’s latest blog post: http://jacobian.org/writing/gpl-questions/
Oct 2nd

ZaN

This works really cool to me, congratulations for this little script that works really good :D (Debian User :D), may a post it on some anime forums man???

Thxs so much this helped me a lot :D
Oct 2nd

Taoufix

@ZaN: “may a post it on some anime forums man?”
Sure, as long as you link to this post :)
Oct 3rd

ZaN

@Taoufix “Sure, as long as you link to this post :)”
Of course I’ll do that :D thxs so much man :D and again, good job with it :D
Nov 14th

Arfy

Now, where is a utility to PUT the crc into the filename? The internet doesn’t seem to have one of those for Linux. Either that or Google, Yahoo, et al. are failing me.
Feb 20th

werefrog

Hi,
Fist, I must confess that I’m totally noob, I’m using Windows and my english is bad :)
Still reading? To be short, I obtain far better performance using binascii.crc32 instead of zlib.crc32 (wathever the file sizes).
Is there something wrong using it?
Regards
Feb 20th

Xiong Chiamiov

Hmm, that’s odd. It seems that most people are getting faster speeds from zlib than binascii[0][1]; there might be something peculiar to the Windows implementation.

0: http://www.mail-archive.com/python-3000@python.org/msg12444.html
1: http://text.maze.io/2009/10/15/generating-crc32-in-python
Feb 27th

werefrog

Thank you for your quick reply and for the links

According what I read about zlib vs binascii performance issue, the quality of zilb compil is a “factor”.

I uninstalled all my python stuff then (after the usual reboot :) installed the active python distribution for x64 including zlib (OS windows seven 64).

Performance are better for zlib but the comparison is still in favor of binascii even in another xp 32bits plateform.

I’ll take a look on freeBSD when I’ll get my machine back (I’m moving a lot) and asked to a friend to try the two on Mac.

Have you tried the two libraries for this script in your machine? (I know the thread’s title include “Linux” but it’s really about python)

Thank you for your time.
Jul 10th

Nekmo

CRC32 with put CRC32

http://pastebin.ca/1897569

## EXAMPLE
nekmo@nekmo-laptop:~/Escritorio$ ./../animeCRC32.py -p *
INFO Nagato1.png -> Nagato1 [92B92065].png
INFO Nagato2.png -> Nagato2 [2168835E].png
INFO Nagato3.png -> Nagato3 [AE446061].png
nekmo@nekmo-laptop:~/Escritorio$ ./../animeCRC32.py -c *
INFO 92B92065 Nagato1 [92B92065].png
INFO 2168835E Nagato2 [2168835E].png
INFO AE446061 Nagato3 [AE446061].png

## ARGS
-c check
-p put

Good Night.

- Nekmo
Sep 5th

whyneus

I guess this comes pretty late but just like to say thanks for the script. Was looking for a script for Linux that did something similar to RapidCRC for Windows.

I guess this is a long shot but if you do ever revise this code, it’d be great if you could add a summary or something at the very end so that you can leave the script churning through a bunch of files and then at the very end have a little summary of what files failed the CRC check.
Apr 1st

Marc Aldorasi

I made a python 3 version available at http://pastebin.com/4tRgTsPQ
It also fixes the problem where checksums would be at the end of the previous line ( like http://ompldr.org/vMXZjZA )
Apr 20th

Joe Gibken

Says the pastebin link has expired. Anybody can repost it again? Thanks :)
May 1st

Nekmo

Here :P http://share.nekmo.com/crc32.py

Features: add CRC32 to files, fixed bugs, logs, python3.x compatible, recursivity, and more :P
May 11th

Veritas

does not work for me:

File “./crc32.py”, line 190

args.check.extendpues e (file_list)

^

SyntaxError: invalid syntax
Sep 1st

John

Fantastic, thank you so much! So easy to use. Perfect!
Jan 25th

tachikoma01

Thanks!
I’ve been looking for a crc32 cli check for so long! Do you know why there’s a cli out of the box for md5 (md5sum) and nothing for crc32?