I have seen rather complex code out there for style sheet swapping in web browsers through JavaScript, and just found out a much simpler way works.
I could have sworn I tried to do real-time style sheet swapping a very long while back and none of my tests turned out satisfactorily, but a friend was just asking me about it, and I was redoing the tests, and it all worked out perfectly in an incredibly easy fashion in IE 6 & 7 and Firefox 2.5 & 3. All that needs to be done is swap the href of the link object pointing to the external style sheet file.
I am a big fan of many SquareSoft games, namely, Final Fantasy 4 (US2), Final Fantasy 6 (US3), and Chrono Trigger. I played all of these on the Super Nintendo many many years ago, and still replay them from time to time through emulator.
I recently recalled that re-releases of these games on the PlayStation consoles included cut scenes, so I decided to look them up. I figured these would be of use to anyone in my boat who is a fan of the old school games but never got to see these.
I included the original links to these videos, which contain author credits, in the title. All videos were found on YouTube, and of course, owned by SquareSoft.
I’ve been waiting to hear this kind of news for years: Domains May Disappear After Search. I’ve often told people for this kind of reason to watch where they are registering domains, as I believe some registrars like register.com are not very scrupulous and would do this exact kind of thing. I personally use GKG for all my domain registration needs, though they have recently ticked me off with a policy I recently ran into in which you can’t modify any information on a domain 6 months after renewing an expired domain with a credit card. Their tech support also isn’t very good, but they have fair prices and excellent domain management interfaces.
Another huge domain problem is “domain tasting” in which domains can be registered and then refunded within a five day grace period. Unethical people will use this to register expired domains and keep them if they get enough hits. After all, domains only really cost 25 cents to register if you are an accredited ICANN (Internet Corporation for Assigned Names and Numbers) registrar, which cost something like $3000 to obtain. This is a big problem if anyone lets their domain expire. Fortunately, some services, like GKG, give you a grace period to reregister your domain after it expires before others can try to claim it.
I made the mistake of trying to watch “Kill Bill”, one of my favorite series of movies, on cable tonight. After suffering through commercials and some horrible edits, I decided it I’d acquire a normal movie copy later on. The edits that werre made to the movie so it could air on TV had me cracking up though. One example was in the long term care hospital the protagonist was staying at with the character “Buck” who “liked to fuck”. He had the word “FUCK” tattooed across one of his hand’s knuckles and his car was named and branded as the “Pussy Wagon”. Since this kind of thing was obviously too much for TV audiences, anytime the word “fuck” was said, it was dubbed over with the word “party”, and his branded car and keychain that said “Pussy Wagon” were overlaid on the screen as “Party Wagon”. It was terribly obtrusive and silly, but it had me laughing at least.
I just now finished watching Disney’s “The Black Cauldron”. While a rather poor example of a Disney animated film, there is one element that really caught my surprise. One of the characters, Gurgi, acted, sounded, and moved just like Golem from Peter Jackson’s rendition of Lord of the Rings. The way Gurgi talked, his inflections, his character’s nature and actions were all pretty much exactly how Golem was portrayed. I’m not necessarily saying Gurgi was stolen from LoTR, or Jackson copied Gurgi alternately, but they are a bit too eerily similar for my speculations.
I’ve been on a bit of a Peter Pan kick lately. It all started with catching Hook a few weeks ago, which I’ve always loved and enjoy watching from time, on the boob tube. After finishing it, I remembered that I was given the original Peter Pan novel for a Christmas when I was around 9 years of age or so, and I decided to pick it up on my next trip to my parents’ house in Dallas. I downloaded all the other official Peter Pan films in the mean time for a watch, as I had never seen them before.
One of the main reasons for this was I was also curious as to how the stories differed in the film versions from the original story, and from each other. I found out they all varied greatly, especially in the tone from the novel, except for Hook, which got it perfect. I’m not going to go into a comparison of the stories here, as that is not really important. All I’d really like to mention about the movies is that the Disney’s 2002 “Return to Neverland” was a rather poor rip off of the Hook plot line, and I didn’t really find it worth it. Disney has really lost it’s flair since The Lion King, IMO. “Walt Disney’s Peter Pan” (February 5, 1953) and “Peter Pan” (2003) however were both well worth it.
The main difference I was referring to between most of the movies and the novel is the heavy presence of a dark and sinister theme in the original book. The Lost Boys were just as cut throat as the pirates, as it mentioned the often battles and killing each other in cold bold, and it even mentioned something to the extent of Peter Pan “thinning out the ranks” of the Lost Boys when their numbers got too large, IIRC. The mermaids were silent killers when they got the chance, and there was also talk of “fairy orgies”. I thought this was all great for a children’s book, as it didn’t concentrate on these aspects, but they were there to give a proper setting. It was a very interesting and fun read, but a far cry from the brilliant status it has been given, IMO. Makes me wonder what all the people out there that complain about Harry Potter would say if they gave this one a read. Oh, the only thing Tinkerbelle pretty much ever says throughout the book is “You ass” :-).
Speaking of Harry Potter, it came as a bit of a shock to me seeing Maggie Smith, who plays Minerva McGonagall in the Harry Potter movies, playing as Granny Wendy in Hook. She did an amazing job at looking decrepit.
One final non-related note… the very briefly overhead Neverland island view shown on Hook really reminded me of my Eternal Realms map.
I had meant to write this post back when I beat “Zelda: Twilight Princess” a few days after it and the Nintendo Wii came out in 2006, but never got around to it, and the idea of writing about a game that came out long ago seemed rather antiquated. The initiative to write this post popped up again though as I just finished replaying “Zelda: Ocarina of Time” (N64).
I have been a really big Zelda fan for a very long time, and have played most of the series. I got to a GameStop ~8 hours, IIRC, before they started preordering the Wii to make sure I could play Twilight Princess as soon as it came out, as I was very anxious to play it. It was a good thing I did too, because when the Wii actually came out, they were next to impossible to acquire. I knew of many people having to wait in lines well over 15 hours to get one soon after the release, and they were still rarities to attain well over a year later.
While I really enjoyed Twilight Princess, I was very frustrated by a rupee and treasure problem. “Zelda” (NES) and “Link to the Past” (SNES) had it right. Whenever you found a secret in those games it was something really worth it, namely, a heart piece (increased your life meter), or often even a new item. Rupees (in game money) were hard earned through slaying enemies, only rarely given in bulk as prizes, and you almost always needed more. As I played through Twilight Princess, I was very frustrated in that almost every secret I found, while hoping for something worth it like a heart pieces, was almost always a mass of rupees. There were at least 50 chests I knew of by the end of the game filled with rupees that I couldn’t acquire because I was almost always maxed out on the amount I could carry. What’s even worse is that the game provided you a means to pretty much directly pinpoint where all heart pieces were. These problems pretty much ruined the enjoyment of the search for secret treasures in the game. You could easily be pointed directly to where all hearts were, new game items were only acquirable as primary dungeon treasures, and the plethora of rupees was next to worthless.
So, as I was replaying Ocarina of Time, I realized how unnecessary rupees were in that game too. There are really only 2 places in the whole game you need rupees to buy important items; one of which is during your very first task within the first few minutes of the game. The only other use for rupees is for a side quest to buy magic beans which takes up a small chunk of your pocket change through the game, but besides that, there is no point to the money system in the game as you never really need it for anything. What’s even more a slap in the face is that one of the primary side quests in the game just rewards you with larger coin purses to carry more rupees, which again, you will never even need to use.
While these games are extremely fun, this game design flaw just irks me. Things like this will never stop me from playing new Zelda games however, or even replaying the old ones from time to time, especially my by far favorite, Link to the Past, as they are all excellent works. I would even call them pieces of art. Miyamoto forever :-).
So I was thinking of a new project that might be fun, useful, and possibly even turn a little profit, but I was talked out of it by a friend due to the true complexity of the prospect past the programming part. The concept isn’t exactly new by a long shot, but my idea for the implementation is, at least I would like to think, novel.
For a very long time, it has been important to be able to prove, without a doubt, that you have the oldest copy of some IP to prove you are the original creator. The usual approach to this is storing copies of the IP at a secure location with the storage time recorded. This is, I am told, very often used in the film industry, as well as many others.
The main downside to this for the subscriber, IMO, is having their IP, which may be confidential, stored by a third party, and entrusting their secrets to an outsider’s security. Of course, if these services are done properly and are ISO certified for non-breachable secure storage, this shouldn’t be a problem as they are probably more secure than anything the user has themselves. One would like to think, though, that entrusting your IP to no one but yourself is the most secure method.
The out-of-house storage method may also require that there be records accessible by others telling that you stored your IP elsewhere, and that it exists, which you may not want known either. This is not always a problem though because some places allow completely anonymous storage.
A large downside for the provider is having to supply and maintain the medium for the secure storage, whether it be vaults for physical property, or hard drives for virtual property.
My solution to this problem, for virtual property anyways, is to not have the provider permanently store the user’s data at all, but provide a means by which the provider can authenticate a set of the user’s data as being unchanged since a certain date. This would be accomplished by hashing the user’s data against a random salt. The salt would be determined by the date and would only be known by the provider.
This would work as follows:
Every day, the server would create a completely random salt string of a fixed length, probably 128 bits. This random salt would be the only thing the server would need to remember and keep secret. This process could also be done beforehand for many days or years.
As the user uploaded the data through a secure connection, the server would process it in chunks, probably 1MB at a time, through the hash function.
The hash function, probably a standard one like MD5, would be slightly modified to multiply the current hash on each block iteration against the daily random salt. The salt would also be updated per upload by a known variable, like multiplying the salt against the upload size, which would be known beforehand, or the exact time of upload.
A signature from a public-key certificate of a combined string of the time of upload and the hash would be calculated.
The user would be returned a confirmation string, which they would need to keep, that contained the time of upload the signature.
Whenever the user wanted to verify their data, they would just have to resend their data and the confirmation string, and the server would return if it is valid or not.
I was thinking the service would be free for maybe 10MB a day. Different account tiers with appropriate fees would be available that would give the user 1 month of access and an amount of upload bandwidth credits, with would roll over each month. Unlimited verifications would also be allowed to account holders, though these uploads would still be applied towards the user’s credits. Verifications without an account would be a nominal charge.
The only thing keeping me from pursuing this idea is that for it to be truly worth it to the end user’s, the processing site and salt tables would have to be secured and ISO certified as such, which would be a lot more costly and trouble than the initial return would provide, I believe, and I don’t have the money to invest in it right now.
I do need to find one of these normal storage services soon for myself soon though. I’ll post again about this when I do.
[edit on 6/15/08 @ 5:04pm] Well, this isn’t exactly the same thing, but a lot like it. http://www.e-timestamp.com
My father’s optometric practice has been using an old DOS database called “Eyecare” since the (I believe) early 80’s. For many years, he has been programming a new, very customized, database up from scratch in Microsoft Access which is backwards compatible with “Eyecare”, which uses a minor variant of FoxPro databases. I’ve been helping him with minor things on it for a number of years, and more recently I’ve been giving a lot more help in getting it secured and migrated from Microsoft Access databases (.mdb) into MySQL.
A recent problem cropped up in that one of the primary tables started crashing Microsoft Access when it was opened (through a FoxPro ODBC driver). Through some tinkering, he discovered that the memo file (.fpt) for the table was corrupted, as trying to view any memo fields is what crashed Access. He asked me to see if I could help in recovering the file, which fortunately I can do at my leisure, as he keeps paper backups of everything for just such circumstances. He keeps daily backups of everything too… but for some reason that’s not an option.
I went about trying to recover it through the easiest means first, namely, trying to open and export the database through FoxPro, which only recovered 187 of the ~9000 memo records. Next, I tried finding a utility online that did the job, and the first one I found that I thought should work was called “FoxFix”, but it failed miserably. There are a number of other Shareware utilities I could try, but I decided to just see how hard it would be to fix myself first.
I opened the memo file up in a HEX editor, and after some very quick perusing and calculations, it was quite easy to determine the format:
So I continued on the path of seeing what I could do to fix the file.
First, I had it jump to the header of each record and just get the record data length, and I very quickly found multiple invalid record lengths.
Next, I had it attempt to fix each of these by determining the real length of the memo by searching for the first null terminator (“\0”) character, but I quickly discovered an oddity. There are weird sections in many of the memo fields in the format BYTE{0,0,0,1,0,0,0,1,x}, which is 2 little endian DWORDS which equal 1, and a final byte character (usually 0).
I added to the algorithm to include these as part of a memo record, and many more original memo lengths then agreed with my calculated memo lengths.
The final thing I did was determine how many invalid (non keyboard) characters there were in the memo data fields. There were ~3500 0x8D characters, which were usually always followed by 0xA, so I assume these were supposed to be line breaks (Windows line breaks are denoted by [0xD/new line/\r],[0xA/carriage return/\n]). There were only 5 other invalid characters, so I just changed these to question marks ‘?’.
Unfortunately, Microsoft Access still crashed when I tried to access the comments fields, so I will next try to just recover the data, tie it to its primary keys (which I will need to determine through the table file [.dbf]), and then rebuild the table. I should be making another post when I get around to doing this.
The following code which “fixes” the table’s memo file took about 2 hours to code up.
//Usually included in windows.h typedef unsigned long DWORD; typedef unsigned char BYTE;
//Memo file structure #pragma warning(disable: 4200) //Remove zero-sized array warning const MemoFileHeadLength=512; const RecordBlockLength=32; //This is actually found in the header at (WORD*)(Start+6) struct MemoRecord //Full structure must be padded at end with \0 to RecordBlockLength { DWORD Type; //Type in little endian, 1=Memo DWORD Length; //Length in little endian BYTE Data[0]; }; #pragma warning(default: 4200)
//Input and output files const char *InFile="EXAM.Fpt.old", *OutFile="EXAM.Fpt";
//Main function void main() { //Read in file const FileSize=6966592; //This should actually be found when the file is opened... FILE* MyFile=fopen(InFile, "rb"); BYTE *MyData=new BYTE[FileSize]; fread(MyData, FileSize, 1, MyFile); fclose(MyFile);
//Start checking file integrity DWORD FilePosition=MemoFileHeadLength; //Where we currently are in the file DWORD RecordNum=0, BadRecords=0, BadBreaks=0, BadChars=0; //Data Counters const DWORD OneInLE=0x01000000; //One in little endian while(FilePosition<FileSize) //Loop until EOF { FilePosition+=sizeof(((MemoRecord*)NULL)->Type); //Advanced passed record type (1=memo) DWORD CurRecordLength=BSWAP(*(DWORD*)(MyData+FilePosition)); //Pull in little endian record size cout << "Record #" << RecordNum++ << " reports " << CurRecordLength << " characters long. (Starts at offset " << FilePosition << ")" << endl; //Output record information
//Determine actual record length FilePosition+=sizeof(((MemoRecord*)NULL)->Length); //Advanced passed record length DWORD RealRecordLength=0; //Actual record length while(true) { for(;MyData[FilePosition+RealRecordLength]!=0 && FilePosition+RealRecordLength<FileSize;RealRecordLength++) //Loop until \0 is encountered { #if 1 //**Check for valid characters might not be needed if(!isprint(MyData[FilePosition+RealRecordLength])) //Makes sure all characters are valid if(MyData[FilePosition+RealRecordLength]==0x8D) //**0x8D maybe should be in ValidCharacters string? - If 0x8D is encountered, replace with 0xD { MyData[FilePosition+RealRecordLength]=0x0D; BadBreaks++; } else //Otherwise, replace with a "?" { MyData[FilePosition+RealRecordLength]='?'; BadChars++; } #endif }
//Check for inner record memo - I'm not really sure why these are here as they don't really fit into the proper memo record format.... Format is DWORD(1), DWORD(1), BYTE(0) if(((MemoRecord*)(MyData+FilePosition+RealRecordLength))->Type==OneInLE && ((MemoRecord*)(MyData+FilePosition+RealRecordLength))->Length==OneInLE /*&& ((MemoRecord*)(MyData+FilePosition+RealRecordLength))->Data[0]==0*/) //**The last byte seems to be able to be anything, so I removed its check { //If inner record memo, current memo must continue ((MemoRecord*)(MyData+FilePosition+RealRecordLength))->Data[0]=0; //**This might need to be taken out - Force last byte back to 0 RealRecordLength+=sizeof(MemoRecord)+1; } else //Otherwise, current memo is finished break; } if(RealRecordLength!=CurRecordLength) //If given length != found length { //Tell the user a bad record was found cout << " Real Length=" << RealRecordLength << endl; CurRecordLength=RealRecordLength; BadRecords++; //getch();
//Update little endian bad record length ((MemoRecord*)(MyData+FilePosition-sizeof(MemoRecord)))->Length=BSWAP(RealRecordLength); }
//Move to next record - Each record, including RecordLength is padded to RecordBlockLength DWORD RealRecordSize=sizeof(MemoRecord)+CurRecordLength; FilePosition+=CurRecordLength+(RealRecordSize%RecordBlockLength==0 ? 0 : RecordBlockLength-RealRecordSize%RecordBlockLength); }
//Tell the user file statistics cout << "Total bad records=" << BadRecords << endl << "Total bad breaks=" << BadBreaks << endl << "Total bad chars=" << BadChars << endl;
//Output fixed data to new file MyFile=fopen(OutFile, "wb"); fwrite(MyData, FileSize, 1, MyFile); fclose(MyFile);
//Cleanup and wait for user keystroke to end delete[] MyData; getch(); }
I am somewhat obsessive about file cleanliness, and like to have everything I do well organized with any superfluous files removed. This especially translates into my source code, and even more so for released source code.
Before I zip up the source code for any project, I always remove the extraneous workspace compilation files. These usually include:
C/C++: Debug & Release directories, *.ncb, *.plg, *.opt, and *.aps
Unfortunately, a new offender surfaced in the form of the Hyrulean Productions icon and Signature File for about pages. I did not want to have to have every source release include those 2 extra files, so I did research into inlining them in the resource script (.rc) file. Resources are just data directly compiled into an executable, and the resource script tells the executable all of these resources and how to compile them in. All my C projects include a resource script for at least the file version, author information, and Hyrulean Productions icon. Anyways, this turned out to be way more of a pain in the butt than intended.
There are 2 ways to load “raw data” (not a standard format like an icon, bitmap, string table, version information, etc) into a resource script. The first way is through loading an external file:
RESOURCEID and RESOURCETYPE are arbitrary and user defined, and it should also be noted to usually have them in caps, as the compilers seem to often be picky about case.
The second way is through inlining the data:
RESOURCEIDRESOURCETYPE BEGIN DATA END
for example:
DakSigSig BEGIN 0x32DA,0x2ACF,0x0306,... END
Getting the data in the right format for the resource script is a relatively simple task.
First, acquire the data in 16-bit encoded format (HEX). I suggest WinHex for this job. On a side note, I have been using WinHex for ages and highly recommend it. It’s one of the most well built and fully featured application suites I know if.
Lastly, convert the straight HEX DATA (“DA32CF2A0603...”) into an array of proper endian hex values (“0x32DA,0x2ACF,0x0306...”). This can be done with a global replace regular expression of “(..)(..)” to “0x$2$1,”. I recommend Editpad Pro for this kind of work, another of my favorite pieces of software. As a matter of fact, I am writing this post right now in it :-).
Here is where the real caveats and problems start falling into place. First, I noticed the resource data was corrupt for a few bytes at a certain location. It turned out to be Visual Studio wanting line lengths in the resource file to be less than ~4175 characters, so I just added a line break at that point.
This idea worked great for the about page signature, which needed to be raw data anyways, but encoding the icon this way turned out to be impossible :-(. Visual Studio apparently requires external files be loaded if you want to use a pre-defined binary resource type (ICON, BITMAP, etc). The simple solution would be to inline the icon as a user defined raw data type, but unfortunately, the Win32 icon loading API functions (LoadIcon, CreateIconFromResource, LoadImage, etc) only seemed to work with properly defined ICONs. I believe the problem here is that when the compiler loads in the icon to include in the executable, it reformats it somewhat, so I would need to know this format. Again, unfortunately, Win32 APIs failed me. FindResource/FindResourceEx wouldn’t let me load the data for ICON types for direct coping (or reverse engineering) :-(. At this point, it wouldn’t be worth my time to try and get the proper format just to inline my Hyrulean Productions icon into resource scripts. I may come back to it later if I’m ever really bored.
This unfortunately brings back a lot of bad old memories regarding Win32 APIs. A lot of the Windows system is really outdated, not nearly robust enough, or just way too obfuscated, and has, and still does, cause me innumerable migraines trying to get things working with their system.
As an example, I just added the first about page to a C project, and getting fonts working on the form was not only a multi-hour long knockdown drag out due to multiple issues, I ended up having to jury rig the final solution in exasperation due to time constraints. I wanted the C about pages to match the VB ones exactly, but font size numbers just wouldn’t conform between the VBGUI designer and Windows GDI (the Windows graphics API), so I just put in arbitrary font size numbers that matched visually instead of trying to find the right conversion process, as the documented font size conversion process was not yielding proper results. This is the main reason VB (and maybe .NET) are far superior in my book when dealing with GUIs (for ease of use at least, not necessarily ability and power). I know there are libraries out that supposedly solve this problem, but I have not yet found one that I am completely happy with, which is why I had started my own fully fledged cross operating systemGUIlibrary a ways back, but it won’t be completed for a long time.
I was just doing my accounting and I noticed I had 3 double-charges on my Capital One credit card that all happened within a 2 day period. I found this to be very odd since I have never been double-charged on any of my credit cards since I started using them 10 years ago when I was 14.
So I went ahead and submitted 2 charge disputes with Capital One, and a third with the other company I saw double-charged. I then finished my accounting, and noticed that the balance showing up on my Capital One did not include those 3 charges. I validated my suspicions by calling up their customer relations department (getting a lady in India) and confirming that the charges only show up once in my account.
I then did my emails to rescind my previous queries into having the double-charges refunded, and also included in the email to Capital One that their web system (or possibly statement system) has an error and needs to be fixed. The double-charges actually weren’t showing up on the same statements. They showed up once (for May 16th and 17th) on my last month’s statement, and then again (May 17th and 19th) on my current month’s statement. Go Figure.
[Edit on 6/13/08] A few days ago, after an annoying downtime on the Capitol One credit card site, I noticed they added a new feature that now shows your latest charges within a certain period of days (15, 30, etc) instead of just the current billing cycle. So I’m pretty sure the above problem was due to them implementing this new system without warning the user or having any indication of the system change in the interface. I do know how annoying change control is, and the problems that come along with implementing new features on websites which may temporarily confuse users, but I’d expect better from a multinational corporation like this. Then again, this isn’t the first time this kind of thing has happened on their website, so I shouldn’t be surprised.
I’ve been thinking for a while that I need to add “about windows” to the executables of all my applications with GUIs. So I first made a test design [left, psd file attached]
Unfortunately, this requires at least 25KB for the background alone, and this is larger than many of my project executables themselves. This is a problem for me, as I like keeping executables small and simple.
I therefore decided to scratch the background and just go with normal “about windows” and my signature in a spiffy font [BlizzardD]: (white background added by web browser for visibility)
The above PNG signature is only 1.66KB, so “yay”, right? Wrong :-(, it quickly occurred to me that XP does not natively support PNG.
My next though is “what about a GIF?” (GIF is the predecessor to PNG, also lossless): (1.82KB)
I remembered that GIF files worked for VB, so I thought that native Windows’ APImight support it too without adding in extra DLLs, but alas, I was wrong. This at least partially solves the problem for me in Visual Basic, but not fully, as GIF does not support translucency, but only 1 color of transparency, so the picture would look horribly aliased (pixilated).
The final solution I decided on is having a small translucency-mask and alpha-blending it and the primary signature color (RGB(6,121,6)) to the “about windows’ ” background.
Since alpha-blending/translucency is an 8 bit value, a gray-scale (also 8 bits per pixel) image is perfect for a translucency mask format for VB: (1.82KB, GIF)
You may note that this GIF is the exact same size as the previous GIF, which makes sense as it is essentially the exact same picture, just with swapped color palettes.
The final hurdle is how to import the picture into C with as little space wasted as possible. The solution to this is to create an easily decompressable alpha-mask (alpha means translucency).
From there, I figured there would be 2 easy formats for compression that would take very little code to decompress:
Number of Transparent Pixels, Number of Foreground Pixels in a Row, List of Foreground Pixel Masks, REPEAT... (This is a form of “Run-length encoding”)
Start the whole image off as transparent, and then list each group of foreground pixels with: X Start Coordinate, Y Start Coordinate, Number of Pixels in a Row, List of Foreground Pixel Masks
It also helped that there were only 16 different alpha-masks, not including the fully transparent mask, so each alpha-mask could be fit within half a byte (4 bits). I only did the first option because I’m pretty sure the second one would be larger because it would take more bits for an x/y location than for a transparent run length number.
Other variants could be used too, like counting the background as a normal mask index and just do straight run length encoding with indexes, but I knew this would make the file much larger for 2 reasons: this would add a 17th alpha-mask which would push index sizes up to 5 bits, and background run lengths are much longer (in this case 6 bits), so all runs would need to be longer (non-background runs are only 3 bits in this case). Anyways, it ended up creating a 1,652bytefile :-).
This could also very easily be edited to input/output 8-bit indexed bitmaps, or full color bitmaps even (with a max of 256 colors, or as many as you wanted with a few more code modifications). If one wanted to use this for normal pictures with a solid background instead of an alpha-mask, just know the words “Transparent” means “Background” and “Translucent” means “Non-Background” in the code.
GIF and PNG file formats actually use similar techniques, but including the code for their decoders would cause a lot more code bloat than I wanted, especially since they [theoretically] include many more compression techniques than just run-length encoding. Programming for specific cases will [theoretically] always be smaller and faster than programming for general cases. On a side note, from past research I’ve done on the JPEG format, along with programming my NES Emulator, Hynes, they [JPEG & NES] share the same main graphical compression technique [grouping colors into blocks and only recording color variations].
//** Double stars denotes changes for custom circumstance [The About Window Mask] #include <windows.h> #include <stdio.h> #include <conio.h>
//Our encoding functions int ErrorOut(char* Error, FILE* HandleToClose); //If an error occurs, output UINT Encode(UCHAR* Input, UCHAR* Output, UINT Width, UINT Height); //Encoding process UCHAR NumBitsRequired(UINT Num); //Tests how many bits are required to represent a number void WriteToBits(UCHAR* StartPointer, UINT BitStart, UINT Value); //Write Value to Bit# BitStart after StartPointer - Assumes more than 8 bits are never written
//Program constants const UCHAR BytesPerPixel=3, TranspMask=255; //24 bits per pixel, and white = transparent background color
//Encoding file header typedef struct { USHORT DataSize; //Data size in bits - **Should be UINT UCHAR Width, Height; //**Should be USHORTs UCHAR TranspSize, TranslSize; //Largest number of bits required for a run length for Transp[arent] and Transl[ucent] UCHAR NumIndexes, Indexes[0]; //Number and list of indexes } EncodedFileHeader;
int main() { UCHAR *InputBuffer, *OutputBuffer; //Where we will hold our input and output data FILE *File; //Handle to current input or output file UINT FileSize; //Holds input and output file sizes
//The bitmap headers tell us about its contents BITMAPFILEHEADER BitmapFileHead; BITMAPINFOHEADER BitmapHead;
//Read in bitmap header and confirm file type File=fopen("AboutWindow-Mask.bmp", "rb"); //Normally you'd read in the filename from passed arguments (argv) if(!File) //Confirm file open return ErrorOut("Cannot open file for reading", NULL); fread(&BitmapFileHead, sizeof(BITMAPFILEHEADER), 1, File); if(BitmapFileHead.bfType!=*(WORD*)"BM" || BitmapFileHead.bfReserved1 || BitmapFileHead.bfReserved2) //Confirm we are opening a bitmap return ErrorOut("Not a bitmap", File);
//Read in the rest of the data fread(&BitmapHead, sizeof(BITMAPINFOHEADER), 1, File); if(BitmapHead.biPlanes!=1 || BitmapHead.biBitCount!=24 || BitmapHead.biCompression!=BI_RGB) //Confirm bitmap type - this code would probably have been simpler if I did an 8 bit indexed file instead... oh well, NBD. **It has also been programmed for easy transition to 8 bit indexed files via the "BytesPerPixel" constant. return ErrorOut("Bitmap must be in 24 bit RGB format", File); FileSize=BitmapFileHead.bfSize-sizeof(BITMAPINFOHEADER)-sizeof(BITMAPFILEHEADER); //Size of the data portion InputBuffer=malloc(FileSize); fread(InputBuffer, FileSize, 1, File); fclose(File);
//Run Encode OutputBuffer=malloc(FileSize); //We should only ever need at most FileSize space for output (output should always be smaller) memset(OutputBuffer, 0, FileSize); //Needs to be zeroed out due to how writing of data file is non sequential FileSize=Encode(InputBuffer, OutputBuffer, BitmapHead.biWidth, BitmapHead.biHeight); //Encode the file and get the output size
//Write encoded data out File=fopen("Output.msk", "wb"); fwrite(OutputBuffer, FileSize, 1, File); fclose(File); printf("File %d written with %d bytes\n", 1, FileSize);
//Free up memory and wait for user input free(InputBuffer); free(OutputBuffer); getch(); //Pause for user input return 0; }
int ErrorOut(char* Error, FILE* HandleToClose) //If an error occurs, output { if(HandleToClose) fclose(HandleToClose); printf("%s\n", Error); getch(); //Pause for user input return 1; }
UINT Encode(UCHAR* Input, UCHAR* Output, UINT Width, UINT Height) //Encoding process { UCHAR Indexes[256], NumIndexes, IndexSize, RowPad; //The index re-mappings, number of indexes, number of bits an index takes in output data, padding at input row ends for windows bitmaps USHORT TranspSize, TranslSize; //Largest number of bits required for a run length for Transp[arent] (zero) and Transl[ucent] (non zero) - should be UCHAR's, but these are used as explained under "CurTranspLen" below UINT BitSize, x, y, ByteOn, NumPixels; //Current output size in bits, x/y coordinate counters, current byte location in Input, number of pixels in mask
//Calculate some stuff NumPixels=Width*Height; //Number of pixels in mask RowPad=4-(Width*BytesPerPixel%4); //Account for windows DWORD row padding - see declaration comment RowPad=(RowPad==4 ? 0 : RowPad);
{ //Do a first pass to find number of different mask values, run lengths, and their encoded values const UCHAR UnusedIndex=255; //In our index list, unused indexes are marked with this constant USHORT CurTranspLen, CurTranslLen; //Keep track of the lengths of the current transparent & translucent runs - TranspSize and TranslSize are temporarily used to hold the maximum run lengths //Zero out all index references and counters memset(Indexes, UnusedIndex, 256); NumIndexes=0; TranspSize=TranslSize=CurTranspLen=CurTranslLen=0; //Start gathering data for(y=ByteOn=0;y<Height;y++) //Column { for(x=0;x<Width;x++,ByteOn+=BytesPerPixel) //Row { UCHAR CurMask=Input[ByteOn]; //Curent alpha mask if(CurMask!=TranspMask) //Translucent value? { //Determine if index has been used yet if(Indexes[CurMask]==UnusedIndex) //We only need to check 1 byte per pixel as they are all the same for gray-scale **This would need to change if using non 24-bit or non gray-scale { ((EncodedFileHeader*)Output)->Indexes[NumIndexes]=CurMask; //Save mask number in the index header Indexes[CurMask]=NumIndexes++; //Save index number to the mask }
//Length of current transparent run TranspSize=(CurTranspLen>TranspSize ? CurTranspLen : TranspSize); //Max(CurTranspLen, TranspSize) CurTranspLen=0;
//Length of current translucent run CurTranslLen++; } else //Transparent value? { //Length of current translucent run TranslSize=(CurTranslLen>TranslSize ? CurTranslLen : TranslSize); //Max(CurTranslLen, TranslSize) CurTranslLen=0;
//Length of current transparent run CurTranspLen++; } }
ByteOn+=RowPad; //Account for windows DWORD row padding } //Determine number of required bits per value printf("Number of Indexes: %d\nLongest Transparent Run: %d\nLongest Translucent Run: %d\n", NumIndexes, TranspSize=CurTranspLen>TranspSize ? CurTranspLen : TranspSize, //Max(CurTranspLen, TranspSize) TranslSize=CurTranslLen>TranslSize ? CurTranslLen : TranslSize //Max(CurTranslLen, TranslSize) ); IndexSize=NumBitsRequired(NumIndexes); TranspSize=NumBitsRequired(TranspSize); //**This is currently overwritten a few lines down TranslSize=NumBitsRequired(TranslSize); //**This is currently overwritten a few lines down printf("Bit Lengths of - Indexes, Trasparent Run Length, Translucent Run Length: %d, %d, %d\n", IndexSize, TranspSize, TranslSize); }
//**Modify run sizes (custom) - this function could be run multiple times with different TranspSize and TranslSize until the best values are found - the best values would always be a weighted average TranspSize=6; TranslSize=3;
//Start processing data BitSize=(sizeof(EncodedFileHeader)+NumIndexes)*8; //Skip the file+bitmap headers and measure in bits x=ByteOn=0; do { //Transparent run UINT CurRun=0; while(Input[ByteOn]==TranspMask && x<NumPixels && CurRun<(UINT)(1<<TranspSize)-1) //Final 2 checks are for EOF and capping run size to max bit length { x++; CurRun++; ByteOn+=BytesPerPixel; if(x%Width==0) //Account for windows DWORD row padding ByteOn+=RowPad; } WriteToBits(Output, BitSize, CurRun); BitSize+=TranspSize;
//Translucent run CurRun=0; BitSize+=TranslSize; //Prepare to start writing masks first while(x<NumPixels && Input[ByteOn]!=TranspMask && CurRun<(UINT)(1<<TranslSize)-1) //Final 2 checks are for EOF and and capping run size to max bit length { WriteToBits(Output, BitSize+CurRun*IndexSize, Indexes[Input[ByteOn]]); x++; CurRun++; ByteOn+=BytesPerPixel; if(x%Width==0) //Account for windows DWORD row padding ByteOn+=RowPad; } WriteToBits(Output, BitSize-TranslSize, CurRun); //Write the mask before the indexes BitSize+=CurRun*IndexSize; } while(x<NumPixels);
{ //Output header EncodedFileHeader *OutputHead; OutputHead=(EncodedFileHeader*)Output; OutputHead->DataSize=BitSize-(sizeof(EncodedFileHeader)+NumIndexes)*8; //Length of file in bits not including header OutputHead->Width=Width; OutputHead->Height=Height; OutputHead->TranspSize=(UCHAR)TranspSize; OutputHead->TranslSize=(UCHAR)TranslSize; OutputHead->NumIndexes=NumIndexes; } return BitSize/8+(BitSize%8 ? 1 : 0); //Return entire length of file in bytes }
UCHAR NumBitsRequired(UINT Num) //Tests how many bits are required to represent a number { UCHAR RetNum; _asm //Find the most significant bit { xor eax, eax //eax=0 bsr eax, Num //Find most significant bit in eax mov RetNum, al } return RetNum+((UCHAR)(1<<RetNum)==Num ? 0 : 1); //Test if the most significant bit is the only one set, if not, at least 1 more bit is required }
void WriteToBits(UCHAR* StartPointer, UINT BitStart, UINT Value) //Write Value to Bit# BitStart after StartPointer - Assumes more than 8 bits are never written { *(WORD*)(&StartPointer[BitStart/8])|=Value<<(BitStart%8); }
The code to decompress the alpha mask in C is as follows: (Shares some header information with above code)
//Decode void Decode(UCHAR* Input, UCHAR* Output); //Decoding process UCHAR ReadBits(UCHAR* StartPointer, UINT BitStart, UCHAR BitSize); //Read value from Bit# BitStart after StartPointer - Assumes more than 8 bits are never read UCHAR NumBitsRequired(UINT Num); //Tests how many bits are required to represent a number --In Encoding Code--
int main() { //--Encoding Code-- UCHAR *InputBuffer, *OutputBuffer; //Where we will hold our input and output data FILE *File; //Handle to current input or output file UINT FileSize; //Holds input and output file sizes
//The bitmap headers tell us about its contents //Read in bitmap header and confirm file type //Read in the rest of the data //Run Encode //Write encoded data out //--END Encoding Code--
/* //If writing back out to a 24 bit windows bitmap, this adds the row padding back in File=fopen("output.bmp", "wb"); fwrite(&BitmapFileHead, sizeof(BITMAPFILEHEADER), 1, File); fwrite(&BitmapHead, sizeof(BITMAPINFOHEADER), 1, File); fwrite(O2, BitmapFileHead.bfSize-sizeof(BITMAPINFOHEADER)-sizeof(BITMAPFILEHEADER), 1, File);*/
//Free up memory and wait for user input --In Encoding Code-- return 0; }
//Decoding void Decode(UCHAR* Input, UCHAR* Output) //Decoding process { EncodedFileHeader H=*(EncodedFileHeader*)Input; //Save header locally so we have quick memory lookups UCHAR Indexes[256], IndexSize=NumBitsRequired(H.NumIndexes); //Save indexes locally so we have quick lookups, use 256 index array so we don't have to allocate memory UINT BitOn=0; //Bit we are currently on in reading memcpy(Indexes, ((EncodedFileHeader*)Input)->Indexes, 256); //Save the indexes Input+=(sizeof(EncodedFileHeader)+H.NumIndexes); //Start reading input past the header
//Unroll/unencode all the pixels do { UINT i, l; //index counter, length (transparent and then index) //Transparent pixels memset(Output, TranspMask, l=ReadBits(Input, BitOn, H.TranspSize)*BytesPerPixel); Output+=l;
//Translucent pixels l=ReadBits(Input, BitOn+=H.TranspSize, H.TranslSize); BitOn+=H.TranslSize; for(i=0;i<l;i++) //Write the gray scale out to the 3 pixels, this should technically be done in a for loop, which would unroll itself anyways, but this way ReadBits+index lookup is only done once - ** Would need to be in a for loop if not using gray-scale or 24 bit output Output[i*BytesPerPixel]=Output[i*BytesPerPixel+1]=Output[i*BytesPerPixel+2]=Indexes[ReadBits(Input, BitOn+i*IndexSize, IndexSize)]; Output+=l*BytesPerPixel; BitOn+=l*IndexSize; } while(BitOn<H.DataSize);
/* { //If writing back out to a 24 bit windows bitmap, this adds the row padding back in UINT i; UCHAR RowPad=4-(H.Width*BytesPerPixel%4); //Account for windows DWORD row padding RowPad=(RowPad==4 ? 0 : RowPad); Output-=H.Width*H.Height*BytesPerPixel; //Restore original output pointer for(i=H.Height;i>0;i--) //Go backwards so data doesn't overwrite itself memcpy(Output+(H.Width*BytesPerPixel+RowPad)*i, Output+(H.Width*BytesPerPixel)*i, H.Width*BytesPerPixel); }*/ }
UCHAR ReadBits(UCHAR* StartPointer, UINT BitStart, UCHAR BitSize) //Read value from Bit# BitStart after StartPointer - Assumes more than 8 bits are never read { return (*(WORD*)&StartPointer[BitStart/8]>>BitStart%8)&((1<<BitSize)-1); }
Of course, I added some minor assembly and optimized the decoder code to get it from 335 to 266 bytes, which is only 69 bytes less , but it’s something (measured using my Small project). There is no real reason to include it here, as it’s in many of my projects and the included C file for this post.
//---- Put in main function above "//Free up memory and wait for user input" ---- printf(CheckDecode(InputBuffer, O2, BitmapHead.biWidth, BitmapHead.biHeight) ? "good" : "bad");
From there, it just has to be loaded into a bit array for manipulation and set back a bitmap device context, and it’s done! VB Code: (Add the signature GIF as a picture box where it is to show up and set its “Visible” property to “false” and “Appearance” to “flat”)
'Swap in and out bits Private Declare Function GetDIBits Lib "gdi32" (ByVal aHDC As Long, ByVal hBitmap As Long, ByVal nStartScan As Long, ByVal nNumScans As Long, lpBits As Any, lpBI As BITMAPINFOHEADER, ByVal wUsage As Long) As Long Private Declare Function SetDIBitsToDevice Lib "gdi32" (ByVal hdc As Long, ByVal x As Long, ByVal y As Long, ByVal dx As Long, ByVal dy As Long, ByVal SrcX As Long, ByVal SrcY As Long, ByVal Scan As Long, ByVal NumScans As Long, Bits As Any, BitsInfo As BITMAPINFOHEADER, ByVal wUsage As Long) As Long lpBits As Any, lpBitsInfo As BITMAPINFOHEADER, ByVal wUsage As Long, ByVal dwRop As Long) As Long Private Type RGBQUAD b As Byte g As Byte r As Byte Reserved As Byte End Type Private Type BITMAPINFOHEADER '40 bytes biSize As Long biWidth As Long biHeight As Long biPlanes As Integer biBitCount As Integer biCompression As Long biSizeImage As Long biXPelsPerMeter As Long biYPelsPerMeter As Long biClrUsed As Long biClrImportant As Long End Type Private Const DIB_RGB_COLORS = 0 ' color table in RGBs
'Prepare colors Private Declare Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" (Destination As Any, Source As Any, ByVal Length As Long) Private Declare Function GetBkColor Lib "gdi32" (ByVal hdc As Long) As Long
Public Sub DisplaySignature(ByRef TheForm As Form) 'Read in Signature Dim BitmapLength As Long, OutBitmap() As RGBQUAD, BitInfo As BITMAPINFOHEADER, Signature As PictureBox Set Signature = TheForm.Signature BitmapLength = Signature.Width * Signature.Height ReDim OutBitmap(0 To BitmapLength - 1) As RGBQUAD With BitInfo .biSize = 40 .biWidth = Signature.Width .biHeight = -Signature.Height .biPlanes = 1 .biBitCount = 32 .biCompression = 0 'BI_RGB .biSizeImage = .biWidth * 4 * -.biHeight End With GetDIBits Signature.hdc, Signature.Image, 0, Signature.Height, OutBitmap(0), BitInfo, DIB_RGB_COLORS
'Alpha blend signature Dim i As Long, Alpha As Double, BackColor As RGBQUAD, ForeColor As RGBQUAD, OBC As Long, OFC As Long OFC = &H67906 OBC = GetBkColor(TheForm.hdc) CopyMemory BackColor, OBC, 4 CopyMemory ForeColor, OFC, 4 For i = 0 To BitmapLength - 1 Alpha = 1 - (CDbl(OutBitmap(i).r) / 255) OutBitmap(i).r = ForeColor.r * Alpha + BackColor.r * (1 - Alpha) OutBitmap(i).g = ForeColor.g * Alpha + BackColor.g * (1 - Alpha) OutBitmap(i).b = ForeColor.b * Alpha + BackColor.b * (1 - Alpha) Next i
SetDIBitsToDevice TheForm.hdc, Signature.Left, Signature.Top, Signature.Width, Signature.Height, 0, 0, 0, Signature.Height, OutBitmap(0), BitInfo, DIB_RGB_COLORS TheForm.Refresh End Sub
//Prepare to decode signature //const UCHAR BytesPerPixel=4, TranspMask=255; //32 bits per pixel (for quicker copies and such - variable not used due to changing BYTE*s to DWORD*s), and white=transparent background color - also not used anymore since we directly write in the background color //Load data from executable HGLOBAL GetData=LoadResource(NULL, FindResource(NULL, "DakSig", "Sig")); //Load the resource from the executable BYTE *Input=(BYTE*)LockResource(GetData); //Get the resource data
//Prepare header and decoding data UINT BitOn=0; //Bit we are currently on in reading EncodedFileHeader H=*(EncodedFileHeader*)Input; //Save header locally so we have quick memory lookups DWORD *Output=Signature=new DWORD[H.Width*H.Height]; //Allocate signature memory
//Prepare the index colors DWORD Indexes[17], IndexSize=NumBitsRequired(H.NumIndexes); //Save full color indexes locally so we have quick lookups, use 17 index array so we don't have to allocate memory (since we already know how many there will be), #16=transparent color DWORD BackgroundColor=GetSysColor(COLOR_BTNFACE), FontColor=0x067906; BYTE *BGC=(BYTE*)&BackgroundColor, *FC=(BYTE*)&FontColor; for(UINT i=0;i<16;i++) //Alpha blend the indexes { float Alpha=((EncodedFileHeader*)Input)->Indexes[i] / 255.0f; BYTE IndexColor[4]; for(int n=0;n<3;n++) IndexColor[n]=(BYTE)(BGC[n]*Alpha + FC[n]*(1-Alpha)); //IndexColor[3]=0; //Don't really need to worry about the last byte as it is unused Indexes[i]=*(DWORD*)IndexColor; } Indexes[16]=BackgroundColor; //Translucent background = window background color
//Unroll/unencode all the pixels Input+=(sizeof(EncodedFileHeader)+H.NumIndexes); //Start reading input past the header do { UINT l; //Length (transparent and then index) //Transparent pixels memsetd(Output, Indexes[16], l=ReadBits(Input, BitOn, H.TranspSize)); Output+=l;
//Translucent pixels l=ReadBits(Input, BitOn+=H.TranspSize, H.TranslSize); BitOn+=H.TranslSize; for(i=0;i<l;i++) //Write the gray scale out to the 3 pixels, this should technically be done in a for loop, which would unroll itself anyways, but this way ReadBits+index lookup is only done once - ** Would need to be in a for loop if not using gray-scale or 24 bit output Output[i]=Indexes[ReadBits(Input, BitOn+i*IndexSize, IndexSize)]; Output+=l; BitOn+=l*IndexSize; } while(BitOn<H.DataSize);
This all adds ~3.5KB to each VB project, and ~2KB to each C/CPP project. Some other recent additions to all project executables include the Hyrulean Productions icon (~1KB) and file version information (1-2KB). I know that a few KB doesn’t seem like much, but when executables are often around 10KB, it can almost double their size.
While I’m on the topic of project sizes, I should note that I always compress their executables with UPX, a very nifty executable compressor. It would often be more prudent to use my Small project, but I don’t want to complicate my open-source code.
One other possible solution I did not pursue would be to take the original font and create a subset font of it with only the letters (and font size?) I need, and see if that file is smaller. I doubt it would have worked well though.
So I have been having major speed issues with one of our servers. After countless hours of diagnoses, I determined the bottle neck was always I/O (input/output, accessing the hard drive). For example, when running an MD5 hash on a 600MB file load would jump up to 31 with 4 logical CPUs and it would take 5-10 minutes to complete. When performing the same test on the same machine on a second drive it finished within seconds.
Replacing the hard drive itself is a last resort for a live production server, and a friend suggested the drive controller could be the problem, so I confirmed that the drive controller for our server was not on-board (on its own card), and I attempted to convince the company hosting our server of the problem so they would replace the drive controller. I ran my own tests first with an iostat check while doing a read of the main hard drive (cat /etc/sda > /dev/null). This produced steadily worsening results the longer the test went on, and always much worse than our secondary drive. I passed these results on to the hosting company, and they replied that a “badblocks –vv” produced results that showed things looked fine.
So I was about to go run his test to confirm his findings, but decided to check parameters first, as I always like to do before running new Linux commands. Thank Thor I did. The admin had meant to write “badblocks –v” (verbose) and typoed with a double key stroke. The two v’s looked like a w due to the font, and had I ran a “badblocks –w” (write-mode test), I would have wiped out the entire hard drive.
Anyways, the test outputted the same basic results as my iostat test with throughput results very quickly decreasing from a remotely acceptable level to almost nil. Of course, the admin only took the best results of the test, ignoring the rest.
I had them swap out the drive controller anyways, and it hasn’t fixed things, so a hard drive replace will probably be needed soon. This kind of problem would be trivial if I had access to the server and could just test the hardware myself, but that is a price to pay for proper security at a server farm.
First, to find out more about any bash command, use
man COMMAND
Now, a primer on the three most useful bash commands: (IMO) find:
Find will search through a directory and its subdirectories for objects (files, directories, links, etc) satisfying its parameters. Parameters are written like a math query, with parenthesis for order of operations (make sure to escape them with a “\”!), -a for boolean “and”, -o for boolean “or”, and ! for “not”. If neither -a or -o is specified, -a is assumed. For example, to find all files that contain “conf” but do not contain “.bak” as the extension, OR are greater than 5MB:
-maxdepth & -mindepth: only look through certain levels of subdirectories
-name: name of the object (-iname for case insensitive)
-regex: name of object matches regular expression
-size: size of object
-type: type of object (block special, character special, directory, named pipe, regular file, symbolic link, socket, etc)
-user & -group: object is owned by user/group
-exec: exec a command on found objects
-print0: output each object separated by a null terminator (great so other programs don’t get confused from white space characters)
-printf: output specified information on each found object (see man file)
For any number operations, use:
+n
for greater than n
-n
for less than n
n
for exactly than n
For a complete reference, see your find’s man page.
xargs:
xargs passes piped arguments to another command as trailing arguments. For example, to list information on all files in a directory greater than 1MB: (Note this will not work with paths with spaces in them, use “find -print0” and “xargs -0” to fix this)
find -size +1024k | xargs ls -l
Some useful parameters include:
-0: piped arguments are separated by null terminators
-n: max arguments passed to each command
-i: replaces “{}” with the piped argument(s)
So, for example, if you had 2 mirrored directories, and wanted to sync their modification timestamps:
GREP is used to search through data for plain text, regular expression, or other pattern matches. You can use it to search through both pipes and files. For example, to get your number of CPUs and their speeds:
cat /proc/cpuinfo | grep MHz
Some useful parameters include:
-E: use extended regular expressions
-P: use perl regular expression
-l: output files with at least one match (-L for no matches)
-o: show only the matching part of the line
-r: recursively search through directories
-v: invert to only output non-matching lines
-Z: separates matches with null terminator
So, for example, to list all files under your current directory that contain “foo1”, “foo2”, or “bar”, you would use:
grep -rlE "foo(1|2)|bar"
For a complete reference, see your grep’s man page.
And now some useful commands and scripts: List size of subdirectories:
du --max-depth=1
The --max-depth parameter specifies how many sub levels to list. -h can be added for more human readable sizes.
List number of files in each subdirectory*:
#!/bin/bash export IFS=$'\n' #Forces only newlines to be considered argument separators for dir in `find -type d -maxdepth 1` do a=`find $dir -type f | wc -l`; if [ $a != "0" ] then echo $dir $a fi done
and to sort those results
SCRIPTNAME | sort -n -k2
List number of different file extensions in current directory and subdirectories:
If you want to make pre-edit backups, include an extension after “-i” like “-i.orig”
Perform operations in directories with too many files to pass as arguments: (in this example, remove all files from a directory 100 at a time instead of using “rm -f *”)
find -type f | xargs -n100 rm -f
Force kill all processes containing a string:
killall -9 STRING
Transfer MySQL databases between servers: (Works in Windows too)
Some lesser known commands that are useful: screen: This opens up a virtual console session that can be disconnected and reconnected from without stopping the session. This is great when connecting to console through SSH so you don’t lose your progress if disconnected. htop: An updated version of top, which is a process information viewer. iotop: A process I/O (input/output - hard drive access) information viewer. Requires Python ? 2.5 and I/O accounting support compiled into the Linux kernel. dig: Domain information retrieval. See “Diagnosing DNS Problems” Post for more information.
More to come later...
*Anything staring with “#!/bin/bash” is intended to be put into a script.
So I’ve been rather perturbed for a very long time at the 50/50 inbox/outbox limit of stored SMS text messages in all LG cell phones. Other phones have similar limits, like a Samsung I have is limited to 100/50, and it just erases messages when an overflow occurs, as opposed to the nice prompts on my LG VX9800, with its QWERTY keyboard, which I love.
I have done some minor hacking on cell phones and tinkered with the firmware, but without a proper emulator, I would never be able to find out where the 50 cap is set and be able to make a hack for phones could store more.
So today, I was at a Verizon store [unimportant ordeal here] because I got a little bit of water on my LG phone and it was having issues. Immediately after the spill, it had a bunch of problems including the battery thinking it was always charging, buttons on the front side sending two different buttons when pressed, and some other buttons not working. I immediately set to shaking it out at all angles to get most of the water out (which there wasn’t much to begin with...), and then I thoroughly blow dried every opening into the inside circuitry. This fixed everything but the worst problem, signal dropping. Basically, the phone would lose any connection it made after about 5 seconds, so I couldn’t really answer or makes calls. Fortunately I was still able to send and receive SMS messages, but received ones didn’t signal the server they were received, and I kept receiving them over and over and over until a connection finally stayed open long enough to tell the server I got it. So I took it back to the store to see if they could fix it, and all they tried was updating the firmware... but they said I could trade it in for another phone for $50, which I figured from the beginning is what I would have to do, and was a good idea anyways because of this [temporarily down]. So they realized they had no replacements in stock... or at the warehouse... for the VX9800 OR the VX9900, which they said they’d upgrade me too if they couldn’t find and VX9800, and I wanted (yay). So I was told to call back tomorrow and try again. Bleh. Anyways, I was at the store where I found out why this was. Apparently, cell phones start slowing down considerably with too many stored SMSs. I was told of a lady that had come in the previous week with 600+ stored messages and the phone took very long intervals to do anything, and clearing it fixed it.
I know that, on my phone at least, each SMS message is stored as a separate file, so my best guess as to the reason for this problem is that this creates too many entries in the file system for the phone to handle. This seems like a rather silly and trivial problem to work around, but the cell phone manufactures can get away with it, as they have no good competitors that fix problems like this.
This is why we really need open source cell phones. There have been word of open source phones in the works for years... but nothing too solid yet .
So ANYWAYS, I had already started taking a different approach in early January to fix the problem of backing up SMS messages without having to sync them to your computer, which is a rather obnoxious work around. I had been researching and planning to write a BREW application that extracts all SMS messages into a text file on your phone so that you don’t have to worry about the limits, and could download them to your computer whenever you wanted, with theoretically thousands of SMS messages archived on your phone. Unfortunately, as usual, other things took over my time and the project was halted, but I will probably be getting back to it soon.