Tuesday, May 3, 2011

Deleted e-mails coming back from the trash in Evolution

For the past couple of months I have been dealing with a very annoying issue with my e-mail client application, Evolution. When I delete an e-mail, it disappears from the message list, which leads me to believe that it has been deleted. If I change to another folder or view and return to the folder where I deleted a message, or if I just wait long enough, the deleted message reappears! I then delete the message a second time and from what I can tell, the second delete does work. This was happening in all of my IMAP folders for a specific account.

As you can imagine, e-mail productivity was non-existent. I found myself reading the same e-mail multiple times, getting frustrated and re-deleting messages I thought I had read. In many cases I was actually deleting the messages that I intended to delete, but in some cases, the frustrated reaction was resulting in deleting a message that I did not want to delete or that I had not yet responded to.

So, I started searching for a bug report or some resolution that would keep me sane. Unfortunately, I had no luck and kept running out of time to do much about it. Do not get me wrong, I found quite a few topics regarding delete issues dealing with IMAP accounts but they all seemed to be configuration issues -- Users wanted to delete a message in Evolution and couldn't understand why it was still showing up in the "All Mail" folder on the server. Or they were attempting to expunge their trash folder only to find that the messages still remained on their e-mail server. In my case, the issue was that e-mails required two deletes to actually get them into the trash folder.

I write the post with optimism, as for today, I stumbled across an Ubuntu forums post that identified a problem different then mine, but similar in nature. My search led me to [SOLVED] Evolution Can not delete emails posted by one bigwoof who had the intelligence to fix his own problem and post the resolution. His issue seemed to be unrelated to deleting e-mails so I am not really sure what led him to the choice of topic but what he was seeing felt similar to what I was potentially dealing with. In his case, there were e-mails showing up in the list that were no longer on the server. I am guessing that he couldn't simply delete them because he was getting an error saying the message could not be found on the server. Additionally, I am using Evolution 2.32.2 while he indicates he upgraded to 2.30. But in my case, the issue seemed to be wacky folder display of some kind. In addition, I have Evolution configured to use a remote folder for the Trash folder. So, when I delete a message, I believe Evolution is actually moving the message to the remote folder. I am not sure if it is also marking it as deleted or not and have not really cared to much about how it is implemented because it had worked for so long.

So I used his solution. In the case of Evolution 2.32.2 my IMAP specific folder was located in a different place, so here is what I did:

  1. Shutdown Evolution
  2. Open my Home folder using Nautilus
  3. Enable Show Hidden Files by pressing CTRL+H or selecting View -> Show Hidden Files
  4. Open the folder named .local
  5. Open the folder named share
  6. Open the folder named evolution
  7. Open the folder named mail
  8. Open the folder named imap
  9. Open the folder which uses the user name and host name for my affected IMAP account, loleary@mail.somewhere.com
  10. Right-click on folders.db and select Rename...
  11. Append .old to the existing file name so it becomes folders.db.old
  12. Launch Evolution
  13. Wait a really long time for it to synchronize all of my mail for offline reading

For the command-line users or the ones who like to enter the full folder path into Nautilus' location bar, it is /home//.local/share/evolution/mail/imap/. From a command-line, it would look like:

mv ~/.local/share/evolution/mail/imap/loleary@mail.somewhere.com/folders.db \
~/.local/share/evolution/mail/imap/loleary@mail.somewhere.com/folders.db.old

I will also note that once you are confident that you have not broken Evolution or your e-mail, feel free to delete the .old folder created as it is no longer useful.

Hope that may help someone sort out other Evolution folder issues that may arise. This should probably be logged as a bug but seeing that I do not know what the real cause is or know what I can do to reproduce it, I don't feel I have enough information to say it really is a bug. Additionally, I only did this a few minutes ago and it seems to have fixed my issue but Evolution is still downloading e-mail for offline reading. Which means, once it is done, maybe those old messages I deleted will show back up! I sure hope not!


UPDATE: Well, wouldn't you know it. I spoke too soon. The multiple delete issue has returned. I probably should have waited before posting this but maybe it will still serve a purpose and maybe someone can steer me in the right direction.

Sunday, May 1, 2011

Is Our E-mail Safe?

Over the past few years it has become evident that we are taking our online safety into our own hands. Some users are undereducated in protecting themselves and their precious data. Some users rely on Internet security software, anti-virus software, anti-spam software, and pop-up blockers to protect them. Some users take for granted what it is they actually need to protect.

I am not a security expert and any information provided here should be viewed as an opinion or observation and is not intended to provide answers or solutions. It is only to remind all of us that we must act more responsible when on-line and take everything we see and hear on-line with skepticism.

This post was inspired by the recent mass e-mail breach reported by Epsilon back on March 31, 2011. This sort of thing happens quite frequently but normally goes without a major fanfare because the information that was leaked or the clients that were involved are not in the masses. In this case, the story picked up national attention because of the clients involved and the amount of data that was retrieved. I will not go into details of the story as you can find all the true and false information on-line. The point is that even when you fully trust someone with a secret, there is always the possibility that the trust will be violated. In many cases, not deliberately, but we are all human and can be misguided or tricked into doing or saying something that we did not intend.

With this most recent episode, supposedly only e-mail addresses and names slipped out. What this means to many of us is that we will start getting more junk e-mail. Hurray! Just what I was needing! This is why I keep multiple e-mail accounts. One is truly personal and if anyone gives it out without my permission I will need to reevaluate my relationship with said person. Another e-mail address is semi-personal. It is the one I give to banks, companies, or legitimate online forums as my e-mail address. I do not like to get junk e-mail in my semi-personal e-mail box, but if I do it is not the end of the world. I then have a third e-mail account that I will give out when I do not really want to hear from the person who is requesting it. I do not check this e-mail account unless I have specifically asked someone or something to e-mail me at that address. This is the e-mail address I will use when I must provide an e-mail address to get something I need.

I can live with the junk e-mail that will most likely result from these recent e-mail address leaks. The real danger of this type of leak is posers or phishing attempts. These are the type of things that even educated Internet users can fall victim to.

What happens is that an e-mail that looks very official comes from a company that you do business with. The e-mail will lead you to believe that something may be wrong with your account or even lead you to believe nothing is wrong with your account. They may simply say, "We are writing to inform you of a recent security breach at one of the advertisers we do business with. Although none of your account information was stored by the third-party and we have no reason to believe your account is at risk, we have taken steps to improve security of our web site. At this time, we ask that you follow the below link to update your security settings. As a reminder, we will never ask you to provide personal or security information via e-mail." Although this e-mail may be legitimate, there is no way to know by simply looking at the e-mail, the sender's address, or the e-mail's headers.

The design of the Internet and the protocols used by e-mail was not intended to be safe and secure. It is similar to the United States Postal Service. A letter is written, placed into an envelope and addressed to a recipient. In addition, a return address may or may not be written on the envelope. Postage is applied to the envelope and it is turned over to a mail carrier. Although the United States Postal Service works very hard to protect these letters from being received or breached by unintended parties while they are carried from the sender to the recipient, there is no way to know that the sender is really who they say they are. Sure, you may be able to identify which postal facility the letter originated from by using the post mark, but this too is meaningless as a means to identify the sender.

What this means is that you should never trust anything received via e-mail. Regardless if you know the sender or not. Many of us hear that we should not trust e-mails from senders that we do not recognize, but really, we shouldn't trust e-mails from anyone! Especially the people we know! We should really be asking, "Is that link Mom sent me actually going to infect me with a virus?" "Did Mom really send me this e-mail or has her computer been infected by a worm that has accessed her address book and e-mail account?"

If you receive an e-mail that asks you to do something pertaining to one of your accounts, the best thing to do is to ignore it or contact the sender using information outside of the e-mail. For example, if this is for a credit card account, call the credit card company using the phone number listed on the back of your card or on an old statement. Or go to their website using the web address printed on the back of the credit card or on an old statement.

The harm in a link

The link displayed in the e-mail says one thing but can actually be taking you to a different site or performing an unintended action.

For example, the link appears to be a valid address to your bank or an online store but could actually be taking you to a completely different site which is not affiliated with your bank or the online store you are doing business with. The "different" site's address will look very similar to the legal site address but its not. This is because some letters and numbers can easily be switched around and the human brain fixes it without you even realizing it. This is how many optical illusions work. Once you go to the fake web site, it looks just like the real thing. You are then tricked into providing your account information such as user name, password, account number, or other information that can be used to access personal information. This is what we have come to know as the Phishing scam.

Additionally, the process of clicking on the link can result in you confirming you are real and potentially result in additional junk e-mail or spam. This is because the link may seem harmless but actually contains a URL parameter that identifies your e-mail address which then tells the sender that your e-mail address is valid and active. This is how the unsubscribe link in many junk e-mails work. Which then results in continued junk e-mails plus a whole lot more. By clicking a link in an e-mail, you are indicating that the e-mail address it was sent to is valid and in use. Therefore, making your e-mail address more valuable to those who are selling it to advertisers, spammers, and other scam artists.

How many words is a picture really worth?

Images displayed in HTML e-mail can actually be reporting receipt of the e-mail. This is similar to clicking on a link in an e-mail. Just by downloading the image, you may actually be sending information to someone on the Internet letting them know that your e-mail address is active and that you received and opened the message. This makes your e-mail address more valuable to advertisers, spammers, and scam artists. Many e-mail applications disable the ability to display HTML images sent in an e-mail and require you to confirm the download of such images before they are downloaded.

This does not mean you can not send images in an e-mail. When you send images in an e-mail, you are normally attaching a physical file to the e-mail. What this pertains to is an image link that is placed in the e-mail body of a message. As a link, the real image isn't downloaded until the e-mail is viewed in an HTML viewer. At that point, your e-mail application goes out onto the Internet and grabs the image from some remote location. It is possible that the link included information that identifies your e-mail address and therefore tells someone out there that your e-mail address is active and more valuable.

The old fashioned attachment

The attachment, although seemingly harmless, is not what it has been advertised as. That e-mail you get from your bank that says "Your latest statement, with personal identifiable information removed, is attached." and indicates that it is a PDF could actually be named "Statement.PDF.exe" and by opening it, you are actually infecting yourself with a virus or malicious software. This can also be true for those cute photos that your sister sends around. They may appear to be JPEG or JPG images and actually display an image when you open them, but in reality, they are executable files that are infesting your computer with a virus or malicious code.

Over time, users have become educated on file types and extensions. They know not to open e-mails with a .EXE or .CMD or .BAT extension. But this isn't good enough. First off, there are many other file formats that can contain malicious code. Additionally, file extensions can be hidden by naming technique or by the e-mail client application itself. It could simply be chopping off the stuff after the last dot because it thinks it knows what the file type is and displays an icon in the attachment list as an indicator to the user. Later versions of WIndows and Outlook have made this a thing of the past, but keep in mind that people are always trying to find new exploits.

Are you really you?

The sender of the e-mail can be faked. Each e-mail contains detailed message headers that show where it originated and what servers it touched along the way but this information is not validated and can be added or faked by the original sender or by anyone who intercepted the e-mail along its route to your e-mail box.

A legitimate request for information

From time to time we receive e-mails from our bank or other companies we do business with, that remind us that they will never ask us for personal or account information. That is a great reminder but what about e-mails we get from other people? Like our family members? Should they be asking us for personal information? The answer is no! We should never ask anyone for personal or confidential information over e-mail. Even more importantly, we should never send personal or confidential information via e-mail or provide it using a method the e-mail indicates unless it is being done through a mutually trusted method.

Case and point: I could be at the video store renting a movie. I decide to use my debit card to pay for it. Meanwhile, someone near the back of the store takes a high-resolution photo of me swiping my card at the register. Later, they can review the photo and establish my card number, my full name, and who I bank with. Now that they have this information they can do a bit more research using online tools to track down my spouse's name and both our e-mail addresses. A few months later, I may receive an e-mail from my wife that simply says "I meant to ask you last night but I forgot! What is our user name and password for our First Bank of Scams account? I need to confirm a transaction!" Well, there is probably a pretty good chance that I will easily realize that she is setting right next to me and ask her why she is e-mailing me, but what if she wasn't. What if I am away on business and talked to her on the phone last night. I might immediately think that it is a legitimate request. The thing to remember is that it is never a legitimate request. If she wants to ask me for that information via e-mail, I have no problems with it. But I should never provide the information requested by replying to the e-mail or providing it through a means she requested that I am not familiar with, such as sending it as a text message to a mobile number that is similar to hers but not the same. I should call her at a number I am familiar with or tell her to call me. Even if I am comfortable providing her the information over e-mail, I should do so by sending a new message to her e-mail address and not via the reply method.

Stop using e-mail?

I am not saying that we should not use e-mail but when we do, we must understand the risks involved and anything that seems suspicious or tells you that you need to take immediate action may actually not be from who it appears. In situations where we are being asked to do something out of the ordinary or provide sensitive or confidential information, we should use a means unrelated to e-mail to validate the request and the sender.

Over the years, attempts have been made to make e-mail more secure. One method is e-mail signing. The concept of a signature is a means to identify the person who the signature belongs to. The idea is not that the signature says Larry O'Leary but it is the way the signature visually looks and how it is physically made up. The idea is that people who know you or do business with you will recognize your signature and be able to spot a fake signature easily. Unfortunately, this too has proven to be something that most of us can not do as we are not trained in hand writing analysis or forgery detection. With e-mail, the same problem exists. Just because I sign my e-mail by typing Larry O'Leary at the bottom doesn't mean that someone else can't do the same. Because of this, we need a way to digitally sign our e-mails and the stuff we send to each other. One such signing technology is Pretty Good Privacy (PGP). The idea is that a sender uses a private key to sign or even encrypt something they are sending. The recipient has the sender's public key so they can verify the item came from the identified sender and validate that the sender's key is valid and really belongs to the sender. This type of digital signing and encryption is great and can make e-mail a safer means to communicate. The frustrating thing is that it is not in wide adoption. My bank still sends me plain-text and HTML e-mail without any digital signature or means of validating that it really came from them. Furthermore, it still doesn't make the content of the e-mail safe. I must still rely on my own common sense and paranoid tenancy to keep myself safe. Just because I know for certain that the e-mail came from Mom doesn't mean the link she sent me is safe.

Modern e-mail client applications such as Microsoft Outlook, Evolution, and Thunderbird do a great job with their default configuration making your use of e-mail a little safer. They block the loading of images unless you say otherwise. They also make opening attachments a bit more difficult so that you don't accidentally install a nasty virus. They do a fairly good job at detecting spam or phishing attempts and try to keep you from falling a victim. But with all these security practices being implemented all we can do is complain that it is too hard to use them. We end up disabling those annoying "Are you sure you want to do this?" pop-ups because we feel that our actions are considerate and safe. But later we can easily become victim to trickery by being caught off guard. Maybe we are having a bad week or preoccupied with something else going on in our busy life. Or maybe we just ran out of coffee and are having a hard time waking up. It only takes one wrong click of the mouse to land us in "Oops! I didn't mean to do that!"

We also have web mail clients such as Yahoo! Mail, Gmail, and Hotmail. Web mail clients offer us another level of security. In most cases, if we accidentally open an attachment, it has been scanned by an anti-virus scanner and the web mail provider is hopefully keeping their anti-virus software up-to-date. We also benefit from the web mail provider keeping their web mail application up-to-date and helping us protect ourselves. However, this in itself may lead to irresponsible behavior on our part.

The take away is that even with all the steps that others have taken to protect us, we are still responsible for our own actions and any of us can fall victim to an array of unacceptable social behavior and engineering with or without our realization. Just remember, e-mail is much like the telephone. The information you receive from the other end may or may not be true and you may or may not know who is on the other end.

Saturday, April 30, 2011

Going from GnuCash 2.2.9 to 2.4.5

Going from one version of an application to another is no big deal. It normally does not require one to give much thought to their existing data or configuration as it is a common expectation that the application upgrade process will take care of upgrading any required configuration or data changes that are being introduced by the new version of the application. Thus, the process is quite simple and painless.

That being said, it is always important not to take things for granted, lower your expectations, and never assume. So, be sure to make a backup of all important data and configuration before proceeding with any update or upgrade that may alter the features of the application or data the application reads and writes. I have known too many people that took this for granted and had something go wrong with an upgrade. Only to find themselves wishing they had their original data. Keep in mind that if the data is modified by the updated application, it normally will not work in the prior version of the application either. This can be true even if you didn't actually modify the data yourself. In some cases, an updated application will make changes to file formats, data structures, and data formats.

For GnuCash, backing up the application data and configuration is pretty simple when on a UNIX based platform. I can not say the same for Windows as I am not a Windows user but I have heard the process is similar.

To backup my data, all I need to do is locate my GnuCash file(s) that I want to save from a potential disaster. I am actually already doing this via automated system backups as everyone is probably already doing, but then again, there goes that assumption thing. The location of your GnuCash file will depend on where you saved it. You can identify the file name by simply looking at the window's title bar in GnuCash or using the options provided in the GnuCash File menu. Unfortunately, I do not have GnuCash 2.2.9 installed so I do not know for sure how this can be done. But, in most applications, I normally achieve this by using the File -> Save As... option -- which will normally open a dialog at the location of my existing file along with the existing file name in the File Name or Save As text field. All I should need to do is make a note of the directory and file name so that I can later retrieve it from the file system. In my case, I already know where my file is located and named so there is no need for me to dig around to find it. So first, I will create a folder to store my backup in.

  1. Navigate to my Documents folder (or wherever you want to store the backup)
  2. Create a new folder named GnuCash-2.2.9-Backup (or whatever you want to name it)

Now that I have a place to store the backup, I need to copy my GnuCash data file there.

  1. Navigate to the folder that contains my GnuCash data file -- Documents/Finance/Household Accounts
  2. Right-click on the file and select Copy -- In my case, the file is named Household.gnucash
    • Be sure to select Copy! You do not want to move it (Cut) or drag-n-drop it as this defeats the purpose of making a backup copy
  3. Navigate to the backup folder GnuCash-2.2.9-Backup created previously
  4. Right-click on an empty place in the folder and select Paste or select Edit -> Paste from the folder's menu

You should now have a copy of your GnuCash data file stored in your backup folder. We can now move on to configuration. The primary reason for backing up the configuration is so that if anything goes wrong with the update or we find that the upgraded version of GnuCash won't run for us, we can revert back to the older version without having to start all over. This part can become tricky as the process will vary from operating system to operating system. So, I apologize if this gives you difficulty.

  1. Navigate to your home folder
  2. Enable Show Hidden Files if not already
    • This is important because the GnuCash configuration directories are normally hidden
    • In Nautilus, this can be done by pressing CTRL+H or from the View -> Show Hidden Files menu
  3. Right-click on the folder named .gnucash and select Copy
    • Pay special attention to the period (.) in front of gnucash
    • Be sure to select Copy! You do not want to move it (Cut) or drag-n-drop it as this defeats the purpose of making a backup copy
  4. Navigate to the backup folder GnuCash-2.2.9-Backup created previously
  5. Right-click on an empty place in the folder and select Paste or select Edit -> Paste from the folder's menu

We have now made a backup of our GnuCash data file (the file which contains all of the important account data) and our GnuCash configuration data which also includes any custom reports we created and information about books if using that feature of GnuCash. This is probably enough for most of us, however, depending on what features you use from GnuCash, there may be other folders that we should backup just in case. For example, if you are using or ever used the online banking feature of GnuCash, AqBanking would have been in use and contains its own configuration.

  1. Navigate to your home folder
  2. Enable Show Hidden Files if not already
    • This is important because the GnuCash configuration directories are normally hidden
    • In Nautilus, this can be done by pressing CTRL+H or from the View -> Show Hidden Files menu
  3. Right-click on the folder named .aqbanking and select Copy
    • Pay special attention to the period (.) in front of aqbanking
    • Be sure to select Copy! You do not want to move it (Cut) or drag-n-drop it as this defeats the purpose of making a backup copy.
  4. Navigate to the backup folder GnuCash-2.2.9-Backup created previously
  5. Right-click on an empty place in the folder and select Paste or select Edit -> Paste from the folder's menu

There may be other configuration files or elements that I have not identified, so if you want to be really safe, backup/mirror your entire hard drive so that you can simply revert to the backup/mirror in the event that something goes wrong with the upgrade. Also, feel free to comment with other folder and file names that I have excluded so that others who should care about them can.

We are now ready to install the new version of GnuCash. Because I am using a Red Hat based Linux operating system, the upgrade is as simple as going to Add/Remove Software and telling it what I want to install and it does the rest.

  1. Navigate to and select System -> Administration -> Add/Remove Software
  2. In the Search field, select Search by name
  3. In the Search field, enter gnucash
  4. Select the Find button
  5. Place a check next to the Finance management application - gnucash package -- in my case gnucash-2.4.5-1
  6. Click the Apply button
  7. If additional software dependencies are required, click the Continue button
  8. Authenticate as the administrative user
    • If you are the only administrative user, enter your password
    • If there are more then one administrative users, select your user name in the drop-down list and enter your password
    • If there are no administrative users, you will need to enter the root user's password
Once all packages have been downloaded and installed, you are ready to fire up GnuCash.

Strangely enough, when I launched the new version of GnuCash, it told me that my saved reports have been upgraded to the new format. It then treated me as a new user of GnuCash, which I find a bit unsettling. Primarily because it knows that I have saved reports but doesn't seem to know that I have an existing data file that I have been using. No problem though, I simply opened my Household Accounts file by going to File -> Open... and navigating to the folder in which my file is stored and selecting it and clicking Open. Once my file had been opened I could see that all seems to be intact. However, I really won't know until I reconcile some accounts over the next few weeks. Because I am not 100% certain that all went as planned, I will keep my backup in a safe place for the next couple of months.

Friday, January 14, 2011

How to generate encrypted shadow password

I recently had a need to generate some encrypted passwords to store in a Kickstart file. This seemed like it should be a fairly straight forward task but it stumped me for a bit. Basically, I needed to take a plain text password and encrypt it as part of the rootpw or user option in the Kickstart file. My initial search of how to do this lead me to many quick solutions. However, upon closer look, the quick solutions only seemed to point to how to generate an MD5 hashed password. I really wanted a SHA512 hash. So, I continued my search -- And found nothing useful. Do not get me wrong. I found many ways of using PHP, database utilities, optional packages, and even some strange shell scripts that would essentially become utilities of their own. I finally gave up on my search and decided to figure it out. The search wasn't completely a waste of time because the information I gained during my search allowed me to come up with a solution. The solution was based on combining information I read within the crypt() method along with information I found on how to generate MD5 hashes.
Two simple methods, choose one that best fits your need:
python -c 'import crypt; print crypt.crypt("Red Hat", "$6$somesalt$")'

perl -e 'print crypt("Red Hat", "\$6\$somesalt\$") . "\n"'

In both methods, the first parameter value of Red Hat is my plain text password that I want to generate a hash for. The second parameter is the salt needed by the crypt() function. The salt is what is very important here. It must start out with $6$ followed by a salt value, followed by a $. The $6 tells the crypt function to use SHA512 and the second $ serves as a delimiter between the hash method and the actual salt. The last $ is needed because it is what appears in the shadow password as a delimiter between the salt and the actual hash. So, if we looked at a shadow password:
$6$somesalt$rkDq7Az4Efjo.0vtRe7E/8UXZNB6.88eMNIaqLdpmULIFkw3gDNPFnjAX4Y9fiGFzVlOyaDr1X.YYUsczCeMH.

The 6 defines the hash algorithm that was used. In this case, SHA512. The somesalt defines the actual salt that is used to generate the hash. The dollar-sign ($) is not used as part of the salt. Finally, the remaining string after the third dollar-sign ($) is the actual hash.
When invoking the crypt() method, essentially the same format applies. Within the salt parameter value, you can define the salt along with the hash algorithm to use. In the case of the perl command, the dollar-signs ($) had to be escaped.

Wednesday, December 16, 2009

Unmounting --rbind mounts in Linux

My latest confusion with using mount and its handy --bind and --rbind options is with unmounting such mounts. It should have been as simple as:

umount /my/mount/point

But when executing such a command I end up with:
umount: /my/mount/point: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))

So the error had me baffled as the mount shouldn't have been in use. I initially thought some process was grabbing a hold of it and causing me some headache. So I used lsof and fuser command to identify who was tying my mount up. It turned out that there were lots of processes tying it up. Here is a more specific example that I was working with:
mount --rbind /home /export/home

I used --rbind because /home contains a couple of mount points that I also wanted to export as part of binding /home to /export/home. Seemed simple enough. Well, after using the umount command and getting the aforementioned error I used lsof and fuser to see that just about every process from /home showed up as using both /home and /export/home.


So, I did a bit more research and found this RFE (although old): https://bugzilla.redhat.com/show_bug.cgi?id=194342


Basically it was talking about recursive binds not being able to be unmounted due to mounts inside the mount having to be unmounted first. So I immediately ran:

umount /export/home/share1
umount /export/home/share2

Yes! No error! So, now:
umount /export/home

No! The error is still there. I couldn't figure this out. I banged my head against the keyboard a few times and then decided to analyse the workaround command provided in the RFE's comments:
mount | awk '{print $3}' | grep ^/dev | sed -e s,/dev,/tmp/dev, | xargs umount
umount /tmp/dev

I plugged /home in for /dev and /export/home for /tmp/dev and executed without the xargs command on the end (I wanted to see what it was going to output):
/export/home
/export/home/share1
/export/home/share2
/export/home/user01/.gvfs

Well wouldn't you know it! I forgot about gvfs. Anyway, that prompted me to throw a quick script together. It is very raw and might not be 100% correct but it gets the job done for me and makes it so I don't have to go back and figure out where all my child-mounts are.


#!/bin/sh

MOUNTS=`mount`
OFS=$IFS

IFS=$'\n'
if [ -d "$1" ]; then
OLDWD="`pwd`"
cd "$1"
rootMount="`pwd`"
cd "$OLDWD"
else
rootMount="${1%/}"
fi

for mount in $MOUNTS; do
umountTgt=`echo $mount | awk '{print $3}'`
if [ "$umountTgt" = "$rootMount" ]; then
mountSrc=`echo $mount | awk '{print $1}'`
umountList=$umountList`mount | awk '{print $3}' | grep ^$mountSrc | sed -e s,$mountSrc,$rootMount, `$'\n'
for umount in $umountList; do
if [ "$umount" != "$rootMount" ]; then
echo "Unmounting $umount..."
/bin/umount "$umount"
fi
done
fi
done

IFS=$OFS

echo "Unmounting $rootMount..."
/bin/umount "$rootMount"

Friday, October 16, 2009

Recovering a deleted file from an Ext4 file system

Well, it seems that no matter how many times it accidentally happens and you tell yourself that you won't make that mistake again it seems that the mistake finds a way of repeating itself.

A while back I was going through some old copies of files from my home directory and getting rid of stuff that was taking up too much disk space. I find it easier to do things from the command-line so there I was. Two terminal windows open... comparing files... seeing what I had extra copies of... removing the old out-of-date copy... I had probably been doing this for a couple of hours and was just about done. I ran across a directory named Larry that I didn't need any more so I simply did a rm -rf Larry and pressed return. About 3 seconds later my stomach jumped up into my throat, sweat started pouring off my forehead, my chest got tight... this was all because I realized that I was in the wrong terminal window and it just so happened that the present work directory that I was in for that terminal window was /home and my user directory happened to be named Larry. I hit CTRL+C and assessed the damage.

Well, as it turned out I couldn't tell that anything important was missing but I knew that there were things that I wouldn't realize were gone until I needed them. So, I found a nice little utility called ext3grep that allowed me to track down the deleted files and restore them. It saved my life! It turned out there was a lot of files deleted and many of them were very important. At that moment I told myself that when using rm -rf I will be more careful.

As a secondary precaution I started doing hourly diff backups of my important directories in case something happened like this again. Well, that was so 2008! Since then I upgraded my OS and haven't gotten around to setting up the hourly diff backups. I am still careful with the rm -rf command but wouldn't you know it!

Today I was creating a new project in Eclipse. I don't like putting all of my projects in the root workspace folder so instead I organize them by type. In this case, my new project was an example EJB but instead of the project being created in workspace/mycoolprojects/ejb/myprojectname it was created in workspace/mycoolprojects/ejb. Not a big deal but now this project's contents was intermixed with all my other project folders. I started by refactoring and moving the project to the correct location but Eclipse complained that the project would overlap another project. So, instead of dealing with it I simply deleted the myprojectname project. When asked if I wanted to delete the contents on disk I simply said yes and ignored the warning saying it couldn't be undone. All seemed fine for about 15 seconds when that voice I hear on occasions called me an idiot and told me how, from Eclipse's perspective, the project was workspace/mycoolprojects/ejb and therefore it wouldn't be surprising if Eclipse just deleted the ejb directory and everything in it. A quick check revealed that, as usual, the voice was right.

Well, I will say that in this case there was only one project in the ejb folder that I was upset about and that was the one that I had been working on just 6 hours earlier. It had not been committed to SCM or backed-up (again, haven't yet setup hourly diff backups). So I then went on the search for how to restore the missing stuff. At the time I didn't remember the utility I used last time this happened so I did a lot of searching. ... probably spent two hours searching. All I could find is a lot of pessimistic answers saying that it was virtually impossible to restore accidentally deleted files on an ext3 or ext4 file system.

Well, I knew this just wasn't true because I knew I had done it before and it actually seemed relatively easy last time. Then finally I found it, ext3grep! It all came back to me! So, I installed the package from the repository and fired it up. Unfortunately it simply returned some strange error about the superblock type not matching some hex value. I then did a bit more searching (now that I knew what I was looking for) and found that ext3grep just won't work with an ext4 file system.

I went back to searching hoping that someone had ported or was working on port of ext3grep to work with ext4 file systems. I found some good hits and a couple of projects that looked promising. And then I stumbled on what appeared to be a solution. extundelete (http://extundelete.sourceforge.net/options.html)! It even had the option that I was looking for, --resotre-directory!

Well, unfortunately the --restore-directory option simply told me how many files that I had lost and basically that I wasn't going to get them back. I was devastated and was now approaching the "spent too much time" milestone. But I decided to give extundelete one more try but this time I used its --restore-file option. Sure enough, it worked! I got a file back! Luckily, I had a directory list of all the contents of my ejb directory which meant that all I had to do was pass the list in to extundelete and to make a long story end, I got all my files back!

Thank you extundelete author(s) and community!!

Tuesday, December 9, 2008

Generating A Random Number Within A Range in Java

The task should be simple. If you search for tips, examples, and tutorials on generating random numbers in Java you will find allot of helpful articles. The problem I ran across was that I needed to generate a random number within a range of user specified numbers. Well, this too is simple as you can use the nextInt(n) method of the Random class to accomplish this, right? Well, I must admit, in most cases you can. In my case, however, I couldn't. See, I needed a random number that could potentially be negative as well as positive. So back to searching and you will find that nextInt(n) should also address my need. It should work something like this:


int randNum = random.nextInt( hiNum - loNum + 1 ) + loNum;

Well, this produce the desired result. I can set hiNum and loNum both to positive numbers and get a pretty even distribution of the range. I can also set hiNum and loNum both to negative numbers and get a pretty even distribution of the range. And most importantly I can set loNum to a negative number and hiNum to a positive number and once again get a pretty even distribution of the range. Seems like the perfect solution, right?


Not for me. The problem with this solution is that nextInt(n) accepts an integer as its parameter. An integer seems to work pretty well until we attempt to set the integer to a value larger than Integer.MAX_VALUE. Why would we ever do this you ask? It is simple, the integer value passed to nextInt(n) represents the range of values you want to generate a random number within. The nextInt(n) method was intended to return a positive random integer. What this means is that as soon as we start adding numbers on the negative side of the field to our integer value range we end up with an unsigned integer. In other words the range of -2 to 2 is 5. The range will include -2, -1, 0, 1, and 2. So the nextInt(n) method keeps working until we hit a range of Integer.MAX_VALUE + 1. Once we do that range gets corrupted for all practical purposes because it becomes a negative value which then results in nextInt(n) throwing an exception.


This can be seen by simply setting loNum and hiNum as follows:


int loNum = Integer.MIN_VALUE / 2;
int hiNum = Integer.MAX_VALUE / 2;

Everything would be fine is we add 1 to loNum or subtract 1 from hiNum but as long as we set them as above, expect nextInt(n) to fail.


I searched for quite some time looking for the solution and all solutions that I found made the same mistake in not taking into consideration the maximum size of an integer. The one thing that I did establish very early is that nextInt(n) will not work for my needs as range needs to be a long. So, here is what I came up with:


long range = ( loNum <> 0 ) ? ( ( (long)hiNum - (long)loNum ) + 1) : ( (long)hiNum - (long)loNum );
int randNum = (int)(long)((random.nextDouble() * (long)(range + 1)) + ( ( loNum <> 0 ? (loNum - 1) : loNum )) );

So, I first calculate a long value and set it as the range. If the range is to include the number 0 I have to add 1 to the range to account for it. This is why we see the loNum <> 0 ? ... : .... Once I have calculated my range using a long value I can call the nextDouble() method from the Random class which will return a number between 0.0 and 1.0. I can then take the random double and multiply it by my range. We add 1 to take into account integer rounding that will occur and result in our hiNum not being achieved due to rounding down. I then need to add loNum to the result so that we get the correct range within an integer value. Again, I have to take into account that 0 may or may not be included in the range of integers and if it isn't I need to subtract 1 from the result.


The most important thing I found when putting this together is the types that are involved. What I initially came up with wasn't working because I failed to case loNum and hiNum to long when I was calculating range resulting in range (although a long) being assigned the bad integer value. Once I worked out my casting issues all seemed to come together. The above now allows me to set loNum to Integer.MIN_VALUE and hiNum to Integer.MAX_VALUE and get expected and acceptable results.