Tuesday, May 3, 2011

Deleted e-mails coming back from the trash in Evolution

For the past couple of months I have been dealing with a very annoying issue with my e-mail client application, Evolution. When I delete an e-mail, it disappears from the message list, which leads me to believe that it has been deleted. If I change to another folder or view and return to the folder where I deleted a message, or if I just wait long enough, the deleted message reappears! I then delete the message a second time and from what I can tell, the second delete does work. This was happening in all of my IMAP folders for a specific account.

As you can imagine, e-mail productivity was non-existent. I found myself reading the same e-mail multiple times, getting frustrated and re-deleting messages I thought I had read. In many cases I was actually deleting the messages that I intended to delete, but in some cases, the frustrated reaction was resulting in deleting a message that I did not want to delete or that I had not yet responded to.

So, I started searching for a bug report or some resolution that would keep me sane. Unfortunately, I had no luck and kept running out of time to do much about it. Do not get me wrong, I found quite a few topics regarding delete issues dealing with IMAP accounts but they all seemed to be configuration issues -- Users wanted to delete a message in Evolution and couldn't understand why it was still showing up in the "All Mail" folder on the server. Or they were attempting to expunge their trash folder only to find that the messages still remained on their e-mail server. In my case, the issue was that e-mails required two deletes to actually get them into the trash folder.

I write the post with optimism, as for today, I stumbled across an Ubuntu forums post that identified a problem different then mine, but similar in nature. My search led me to [SOLVED] Evolution Can not delete emails posted by one bigwoof who had the intelligence to fix his own problem and post the resolution. His issue seemed to be unrelated to deleting e-mails so I am not really sure what led him to the choice of topic but what he was seeing felt similar to what I was potentially dealing with. In his case, there were e-mails showing up in the list that were no longer on the server. I am guessing that he couldn't simply delete them because he was getting an error saying the message could not be found on the server. Additionally, I am using Evolution 2.32.2 while he indicates he upgraded to 2.30. But in my case, the issue seemed to be wacky folder display of some kind. In addition, I have Evolution configured to use a remote folder for the Trash folder. So, when I delete a message, I believe Evolution is actually moving the message to the remote folder. I am not sure if it is also marking it as deleted or not and have not really cared to much about how it is implemented because it had worked for so long.

So I used his solution. In the case of Evolution 2.32.2 my IMAP specific folder was located in a different place, so here is what I did:

  1. Shutdown Evolution
  2. Open my Home folder using Nautilus
  3. Enable Show Hidden Files by pressing CTRL+H or selecting View -> Show Hidden Files
  4. Open the folder named .local
  5. Open the folder named share
  6. Open the folder named evolution
  7. Open the folder named mail
  8. Open the folder named imap
  9. Open the folder which uses the user name and host name for my affected IMAP account, loleary@mail.somewhere.com
  10. Right-click on folders.db and select Rename...
  11. Append .old to the existing file name so it becomes folders.db.old
  12. Launch Evolution
  13. Wait a really long time for it to synchronize all of my mail for offline reading

For the command-line users or the ones who like to enter the full folder path into Nautilus' location bar, it is /home//.local/share/evolution/mail/imap/. From a command-line, it would look like:

mv ~/.local/share/evolution/mail/imap/loleary@mail.somewhere.com/folders.db \
~/.local/share/evolution/mail/imap/loleary@mail.somewhere.com/folders.db.old

I will also note that once you are confident that you have not broken Evolution or your e-mail, feel free to delete the .old folder created as it is no longer useful.

Hope that may help someone sort out other Evolution folder issues that may arise. This should probably be logged as a bug but seeing that I do not know what the real cause is or know what I can do to reproduce it, I don't feel I have enough information to say it really is a bug. Additionally, I only did this a few minutes ago and it seems to have fixed my issue but Evolution is still downloading e-mail for offline reading. Which means, once it is done, maybe those old messages I deleted will show back up! I sure hope not!


UPDATE: Well, wouldn't you know it. I spoke too soon. The multiple delete issue has returned. I probably should have waited before posting this but maybe it will still serve a purpose and maybe someone can steer me in the right direction.

Sunday, May 1, 2011

Is Our E-mail Safe?

Over the past few years it has become evident that we are taking our online safety into our own hands. Some users are undereducated in protecting themselves and their precious data. Some users rely on Internet security software, anti-virus software, anti-spam software, and pop-up blockers to protect them. Some users take for granted what it is they actually need to protect.

I am not a security expert and any information provided here should be viewed as an opinion or observation and is not intended to provide answers or solutions. It is only to remind all of us that we must act more responsible when on-line and take everything we see and hear on-line with skepticism.

This post was inspired by the recent mass e-mail breach reported by Epsilon back on March 31, 2011. This sort of thing happens quite frequently but normally goes without a major fanfare because the information that was leaked or the clients that were involved are not in the masses. In this case, the story picked up national attention because of the clients involved and the amount of data that was retrieved. I will not go into details of the story as you can find all the true and false information on-line. The point is that even when you fully trust someone with a secret, there is always the possibility that the trust will be violated. In many cases, not deliberately, but we are all human and can be misguided or tricked into doing or saying something that we did not intend.

With this most recent episode, supposedly only e-mail addresses and names slipped out. What this means to many of us is that we will start getting more junk e-mail. Hurray! Just what I was needing! This is why I keep multiple e-mail accounts. One is truly personal and if anyone gives it out without my permission I will need to reevaluate my relationship with said person. Another e-mail address is semi-personal. It is the one I give to banks, companies, or legitimate online forums as my e-mail address. I do not like to get junk e-mail in my semi-personal e-mail box, but if I do it is not the end of the world. I then have a third e-mail account that I will give out when I do not really want to hear from the person who is requesting it. I do not check this e-mail account unless I have specifically asked someone or something to e-mail me at that address. This is the e-mail address I will use when I must provide an e-mail address to get something I need.

I can live with the junk e-mail that will most likely result from these recent e-mail address leaks. The real danger of this type of leak is posers or phishing attempts. These are the type of things that even educated Internet users can fall victim to.

What happens is that an e-mail that looks very official comes from a company that you do business with. The e-mail will lead you to believe that something may be wrong with your account or even lead you to believe nothing is wrong with your account. They may simply say, "We are writing to inform you of a recent security breach at one of the advertisers we do business with. Although none of your account information was stored by the third-party and we have no reason to believe your account is at risk, we have taken steps to improve security of our web site. At this time, we ask that you follow the below link to update your security settings. As a reminder, we will never ask you to provide personal or security information via e-mail." Although this e-mail may be legitimate, there is no way to know by simply looking at the e-mail, the sender's address, or the e-mail's headers.

The design of the Internet and the protocols used by e-mail was not intended to be safe and secure. It is similar to the United States Postal Service. A letter is written, placed into an envelope and addressed to a recipient. In addition, a return address may or may not be written on the envelope. Postage is applied to the envelope and it is turned over to a mail carrier. Although the United States Postal Service works very hard to protect these letters from being received or breached by unintended parties while they are carried from the sender to the recipient, there is no way to know that the sender is really who they say they are. Sure, you may be able to identify which postal facility the letter originated from by using the post mark, but this too is meaningless as a means to identify the sender.

What this means is that you should never trust anything received via e-mail. Regardless if you know the sender or not. Many of us hear that we should not trust e-mails from senders that we do not recognize, but really, we shouldn't trust e-mails from anyone! Especially the people we know! We should really be asking, "Is that link Mom sent me actually going to infect me with a virus?" "Did Mom really send me this e-mail or has her computer been infected by a worm that has accessed her address book and e-mail account?"

If you receive an e-mail that asks you to do something pertaining to one of your accounts, the best thing to do is to ignore it or contact the sender using information outside of the e-mail. For example, if this is for a credit card account, call the credit card company using the phone number listed on the back of your card or on an old statement. Or go to their website using the web address printed on the back of the credit card or on an old statement.

The harm in a link

The link displayed in the e-mail says one thing but can actually be taking you to a different site or performing an unintended action.

For example, the link appears to be a valid address to your bank or an online store but could actually be taking you to a completely different site which is not affiliated with your bank or the online store you are doing business with. The "different" site's address will look very similar to the legal site address but its not. This is because some letters and numbers can easily be switched around and the human brain fixes it without you even realizing it. This is how many optical illusions work. Once you go to the fake web site, it looks just like the real thing. You are then tricked into providing your account information such as user name, password, account number, or other information that can be used to access personal information. This is what we have come to know as the Phishing scam.

Additionally, the process of clicking on the link can result in you confirming you are real and potentially result in additional junk e-mail or spam. This is because the link may seem harmless but actually contains a URL parameter that identifies your e-mail address which then tells the sender that your e-mail address is valid and active. This is how the unsubscribe link in many junk e-mails work. Which then results in continued junk e-mails plus a whole lot more. By clicking a link in an e-mail, you are indicating that the e-mail address it was sent to is valid and in use. Therefore, making your e-mail address more valuable to those who are selling it to advertisers, spammers, and other scam artists.

How many words is a picture really worth?

Images displayed in HTML e-mail can actually be reporting receipt of the e-mail. This is similar to clicking on a link in an e-mail. Just by downloading the image, you may actually be sending information to someone on the Internet letting them know that your e-mail address is active and that you received and opened the message. This makes your e-mail address more valuable to advertisers, spammers, and scam artists. Many e-mail applications disable the ability to display HTML images sent in an e-mail and require you to confirm the download of such images before they are downloaded.

This does not mean you can not send images in an e-mail. When you send images in an e-mail, you are normally attaching a physical file to the e-mail. What this pertains to is an image link that is placed in the e-mail body of a message. As a link, the real image isn't downloaded until the e-mail is viewed in an HTML viewer. At that point, your e-mail application goes out onto the Internet and grabs the image from some remote location. It is possible that the link included information that identifies your e-mail address and therefore tells someone out there that your e-mail address is active and more valuable.

The old fashioned attachment

The attachment, although seemingly harmless, is not what it has been advertised as. That e-mail you get from your bank that says "Your latest statement, with personal identifiable information removed, is attached." and indicates that it is a PDF could actually be named "Statement.PDF.exe" and by opening it, you are actually infecting yourself with a virus or malicious software. This can also be true for those cute photos that your sister sends around. They may appear to be JPEG or JPG images and actually display an image when you open them, but in reality, they are executable files that are infesting your computer with a virus or malicious code.

Over time, users have become educated on file types and extensions. They know not to open e-mails with a .EXE or .CMD or .BAT extension. But this isn't good enough. First off, there are many other file formats that can contain malicious code. Additionally, file extensions can be hidden by naming technique or by the e-mail client application itself. It could simply be chopping off the stuff after the last dot because it thinks it knows what the file type is and displays an icon in the attachment list as an indicator to the user. Later versions of WIndows and Outlook have made this a thing of the past, but keep in mind that people are always trying to find new exploits.

Are you really you?

The sender of the e-mail can be faked. Each e-mail contains detailed message headers that show where it originated and what servers it touched along the way but this information is not validated and can be added or faked by the original sender or by anyone who intercepted the e-mail along its route to your e-mail box.

A legitimate request for information

From time to time we receive e-mails from our bank or other companies we do business with, that remind us that they will never ask us for personal or account information. That is a great reminder but what about e-mails we get from other people? Like our family members? Should they be asking us for personal information? The answer is no! We should never ask anyone for personal or confidential information over e-mail. Even more importantly, we should never send personal or confidential information via e-mail or provide it using a method the e-mail indicates unless it is being done through a mutually trusted method.

Case and point: I could be at the video store renting a movie. I decide to use my debit card to pay for it. Meanwhile, someone near the back of the store takes a high-resolution photo of me swiping my card at the register. Later, they can review the photo and establish my card number, my full name, and who I bank with. Now that they have this information they can do a bit more research using online tools to track down my spouse's name and both our e-mail addresses. A few months later, I may receive an e-mail from my wife that simply says "I meant to ask you last night but I forgot! What is our user name and password for our First Bank of Scams account? I need to confirm a transaction!" Well, there is probably a pretty good chance that I will easily realize that she is setting right next to me and ask her why she is e-mailing me, but what if she wasn't. What if I am away on business and talked to her on the phone last night. I might immediately think that it is a legitimate request. The thing to remember is that it is never a legitimate request. If she wants to ask me for that information via e-mail, I have no problems with it. But I should never provide the information requested by replying to the e-mail or providing it through a means she requested that I am not familiar with, such as sending it as a text message to a mobile number that is similar to hers but not the same. I should call her at a number I am familiar with or tell her to call me. Even if I am comfortable providing her the information over e-mail, I should do so by sending a new message to her e-mail address and not via the reply method.

Stop using e-mail?

I am not saying that we should not use e-mail but when we do, we must understand the risks involved and anything that seems suspicious or tells you that you need to take immediate action may actually not be from who it appears. In situations where we are being asked to do something out of the ordinary or provide sensitive or confidential information, we should use a means unrelated to e-mail to validate the request and the sender.

Over the years, attempts have been made to make e-mail more secure. One method is e-mail signing. The concept of a signature is a means to identify the person who the signature belongs to. The idea is not that the signature says Larry O'Leary but it is the way the signature visually looks and how it is physically made up. The idea is that people who know you or do business with you will recognize your signature and be able to spot a fake signature easily. Unfortunately, this too has proven to be something that most of us can not do as we are not trained in hand writing analysis or forgery detection. With e-mail, the same problem exists. Just because I sign my e-mail by typing Larry O'Leary at the bottom doesn't mean that someone else can't do the same. Because of this, we need a way to digitally sign our e-mails and the stuff we send to each other. One such signing technology is Pretty Good Privacy (PGP). The idea is that a sender uses a private key to sign or even encrypt something they are sending. The recipient has the sender's public key so they can verify the item came from the identified sender and validate that the sender's key is valid and really belongs to the sender. This type of digital signing and encryption is great and can make e-mail a safer means to communicate. The frustrating thing is that it is not in wide adoption. My bank still sends me plain-text and HTML e-mail without any digital signature or means of validating that it really came from them. Furthermore, it still doesn't make the content of the e-mail safe. I must still rely on my own common sense and paranoid tenancy to keep myself safe. Just because I know for certain that the e-mail came from Mom doesn't mean the link she sent me is safe.

Modern e-mail client applications such as Microsoft Outlook, Evolution, and Thunderbird do a great job with their default configuration making your use of e-mail a little safer. They block the loading of images unless you say otherwise. They also make opening attachments a bit more difficult so that you don't accidentally install a nasty virus. They do a fairly good job at detecting spam or phishing attempts and try to keep you from falling a victim. But with all these security practices being implemented all we can do is complain that it is too hard to use them. We end up disabling those annoying "Are you sure you want to do this?" pop-ups because we feel that our actions are considerate and safe. But later we can easily become victim to trickery by being caught off guard. Maybe we are having a bad week or preoccupied with something else going on in our busy life. Or maybe we just ran out of coffee and are having a hard time waking up. It only takes one wrong click of the mouse to land us in "Oops! I didn't mean to do that!"

We also have web mail clients such as Yahoo! Mail, Gmail, and Hotmail. Web mail clients offer us another level of security. In most cases, if we accidentally open an attachment, it has been scanned by an anti-virus scanner and the web mail provider is hopefully keeping their anti-virus software up-to-date. We also benefit from the web mail provider keeping their web mail application up-to-date and helping us protect ourselves. However, this in itself may lead to irresponsible behavior on our part.

The take away is that even with all the steps that others have taken to protect us, we are still responsible for our own actions and any of us can fall victim to an array of unacceptable social behavior and engineering with or without our realization. Just remember, e-mail is much like the telephone. The information you receive from the other end may or may not be true and you may or may not know who is on the other end.

Saturday, April 30, 2011

Going from GnuCash 2.2.9 to 2.4.5

Going from one version of an application to another is no big deal. It normally does not require one to give much thought to their existing data or configuration as it is a common expectation that the application upgrade process will take care of upgrading any required configuration or data changes that are being introduced by the new version of the application. Thus, the process is quite simple and painless.

That being said, it is always important not to take things for granted, lower your expectations, and never assume. So, be sure to make a backup of all important data and configuration before proceeding with any update or upgrade that may alter the features of the application or data the application reads and writes. I have known too many people that took this for granted and had something go wrong with an upgrade. Only to find themselves wishing they had their original data. Keep in mind that if the data is modified by the updated application, it normally will not work in the prior version of the application either. This can be true even if you didn't actually modify the data yourself. In some cases, an updated application will make changes to file formats, data structures, and data formats.

For GnuCash, backing up the application data and configuration is pretty simple when on a UNIX based platform. I can not say the same for Windows as I am not a Windows user but I have heard the process is similar.

To backup my data, all I need to do is locate my GnuCash file(s) that I want to save from a potential disaster. I am actually already doing this via automated system backups as everyone is probably already doing, but then again, there goes that assumption thing. The location of your GnuCash file will depend on where you saved it. You can identify the file name by simply looking at the window's title bar in GnuCash or using the options provided in the GnuCash File menu. Unfortunately, I do not have GnuCash 2.2.9 installed so I do not know for sure how this can be done. But, in most applications, I normally achieve this by using the File -> Save As... option -- which will normally open a dialog at the location of my existing file along with the existing file name in the File Name or Save As text field. All I should need to do is make a note of the directory and file name so that I can later retrieve it from the file system. In my case, I already know where my file is located and named so there is no need for me to dig around to find it. So first, I will create a folder to store my backup in.

  1. Navigate to my Documents folder (or wherever you want to store the backup)
  2. Create a new folder named GnuCash-2.2.9-Backup (or whatever you want to name it)

Now that I have a place to store the backup, I need to copy my GnuCash data file there.

  1. Navigate to the folder that contains my GnuCash data file -- Documents/Finance/Household Accounts
  2. Right-click on the file and select Copy -- In my case, the file is named Household.gnucash
    • Be sure to select Copy! You do not want to move it (Cut) or drag-n-drop it as this defeats the purpose of making a backup copy
  3. Navigate to the backup folder GnuCash-2.2.9-Backup created previously
  4. Right-click on an empty place in the folder and select Paste or select Edit -> Paste from the folder's menu

You should now have a copy of your GnuCash data file stored in your backup folder. We can now move on to configuration. The primary reason for backing up the configuration is so that if anything goes wrong with the update or we find that the upgraded version of GnuCash won't run for us, we can revert back to the older version without having to start all over. This part can become tricky as the process will vary from operating system to operating system. So, I apologize if this gives you difficulty.

  1. Navigate to your home folder
  2. Enable Show Hidden Files if not already
    • This is important because the GnuCash configuration directories are normally hidden
    • In Nautilus, this can be done by pressing CTRL+H or from the View -> Show Hidden Files menu
  3. Right-click on the folder named .gnucash and select Copy
    • Pay special attention to the period (.) in front of gnucash
    • Be sure to select Copy! You do not want to move it (Cut) or drag-n-drop it as this defeats the purpose of making a backup copy
  4. Navigate to the backup folder GnuCash-2.2.9-Backup created previously
  5. Right-click on an empty place in the folder and select Paste or select Edit -> Paste from the folder's menu

We have now made a backup of our GnuCash data file (the file which contains all of the important account data) and our GnuCash configuration data which also includes any custom reports we created and information about books if using that feature of GnuCash. This is probably enough for most of us, however, depending on what features you use from GnuCash, there may be other folders that we should backup just in case. For example, if you are using or ever used the online banking feature of GnuCash, AqBanking would have been in use and contains its own configuration.

  1. Navigate to your home folder
  2. Enable Show Hidden Files if not already
    • This is important because the GnuCash configuration directories are normally hidden
    • In Nautilus, this can be done by pressing CTRL+H or from the View -> Show Hidden Files menu
  3. Right-click on the folder named .aqbanking and select Copy
    • Pay special attention to the period (.) in front of aqbanking
    • Be sure to select Copy! You do not want to move it (Cut) or drag-n-drop it as this defeats the purpose of making a backup copy.
  4. Navigate to the backup folder GnuCash-2.2.9-Backup created previously
  5. Right-click on an empty place in the folder and select Paste or select Edit -> Paste from the folder's menu

There may be other configuration files or elements that I have not identified, so if you want to be really safe, backup/mirror your entire hard drive so that you can simply revert to the backup/mirror in the event that something goes wrong with the upgrade. Also, feel free to comment with other folder and file names that I have excluded so that others who should care about them can.

We are now ready to install the new version of GnuCash. Because I am using a Red Hat based Linux operating system, the upgrade is as simple as going to Add/Remove Software and telling it what I want to install and it does the rest.

  1. Navigate to and select System -> Administration -> Add/Remove Software
  2. In the Search field, select Search by name
  3. In the Search field, enter gnucash
  4. Select the Find button
  5. Place a check next to the Finance management application - gnucash package -- in my case gnucash-2.4.5-1
  6. Click the Apply button
  7. If additional software dependencies are required, click the Continue button
  8. Authenticate as the administrative user
    • If you are the only administrative user, enter your password
    • If there are more then one administrative users, select your user name in the drop-down list and enter your password
    • If there are no administrative users, you will need to enter the root user's password
Once all packages have been downloaded and installed, you are ready to fire up GnuCash.

Strangely enough, when I launched the new version of GnuCash, it told me that my saved reports have been upgraded to the new format. It then treated me as a new user of GnuCash, which I find a bit unsettling. Primarily because it knows that I have saved reports but doesn't seem to know that I have an existing data file that I have been using. No problem though, I simply opened my Household Accounts file by going to File -> Open... and navigating to the folder in which my file is stored and selecting it and clicking Open. Once my file had been opened I could see that all seems to be intact. However, I really won't know until I reconcile some accounts over the next few weeks. Because I am not 100% certain that all went as planned, I will keep my backup in a safe place for the next couple of months.

Friday, January 14, 2011

How to generate encrypted shadow password

I recently had a need to generate some encrypted passwords to store in a Kickstart file. This seemed like it should be a fairly straight forward task but it stumped me for a bit. Basically, I needed to take a plain text password and encrypt it as part of the rootpw or user option in the Kickstart file. My initial search of how to do this lead me to many quick solutions. However, upon closer look, the quick solutions only seemed to point to how to generate an MD5 hashed password. I really wanted a SHA512 hash. So, I continued my search -- And found nothing useful. Do not get me wrong. I found many ways of using PHP, database utilities, optional packages, and even some strange shell scripts that would essentially become utilities of their own. I finally gave up on my search and decided to figure it out. The search wasn't completely a waste of time because the information I gained during my search allowed me to come up with a solution. The solution was based on combining information I read within the crypt() method along with information I found on how to generate MD5 hashes.
Two simple methods, choose one that best fits your need:
python -c 'import crypt; print crypt.crypt("Red Hat", "$6$somesalt$")'

perl -e 'print crypt("Red Hat", "\$6\$somesalt\$") . "\n"'

In both methods, the first parameter value of Red Hat is my plain text password that I want to generate a hash for. The second parameter is the salt needed by the crypt() function. The salt is what is very important here. It must start out with $6$ followed by a salt value, followed by a $. The $6 tells the crypt function to use SHA512 and the second $ serves as a delimiter between the hash method and the actual salt. The last $ is needed because it is what appears in the shadow password as a delimiter between the salt and the actual hash. So, if we looked at a shadow password:
$6$somesalt$rkDq7Az4Efjo.0vtRe7E/8UXZNB6.88eMNIaqLdpmULIFkw3gDNPFnjAX4Y9fiGFzVlOyaDr1X.YYUsczCeMH.

The 6 defines the hash algorithm that was used. In this case, SHA512. The somesalt defines the actual salt that is used to generate the hash. The dollar-sign ($) is not used as part of the salt. Finally, the remaining string after the third dollar-sign ($) is the actual hash.
When invoking the crypt() method, essentially the same format applies. Within the salt parameter value, you can define the salt along with the hash algorithm to use. In the case of the perl command, the dollar-signs ($) had to be escaped.