Follow Me On...
Tuesday
Mar042014

Bash Script for monitoring ulimit for multiple processes

Use the script i posted here: https://gist.github.com/blak3r/d470a4254059acca926a

Then pipe it to a file so you can analyze the sockets used by your app overtime.

Saturday
Jan112014

How to determine what's causing: "Error: connect EMFILE" (Node.js)

I came across a bunch of posts on stackoverflow suggesting I use something like graceful-js to solve my problem, or to increase ulimit. Essentially, they’re workarounds. I wanted to know what the root problem was. Here’s the process I came up with. If anyone knows a better way, please let me know in the comments.

What This Error Means

There is a number of file handles a process can have open. Note: that sockets also create a file handle. Once you reach the limit you cannot open anymore and a cryptic error message such as “Error connect EMFILE” will end up in your log file (hopefully). The default limit (at least on my ubuntu system) is 1024.

How To Isolate

This command will output the number of open handles for nodejs processes: lsof -i -n -P | grep nodejs

COMMAND     PID    USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
...
nodejs    12211    root 1012u  IPv4 151317015      0t0  TCP 10.101.42.209:40371->54.236.3.170:80 (ESTABLISHED)
nodejs    12211    root 1013u  IPv4 151279902      0t0  TCP 10.101.42.209:43656->54.236.3.172:80 (ESTABLISHED)
nodejs    12211    root 1014u  IPv4 151317016      0t0  TCP 10.101.42.209:34450->54.236.3.168:80 (ESTABLISHED)
nodejs    12211    root 1015u  IPv4 151289728      0t0  TCP 10.101.42.209:52691->54.236.3.173:80 (ESTABLISHED)
nodejs    12211    root 1016u  IPv4 151305607      0t0  TCP 10.101.42.209:47707->54.236.3.172:80 (ESTABLISHED)
nodejs    12211    root 1017u  IPv4 151289730      0t0  TCP 10.101.42.209:45423->54.236.3.171:80 (ESTABLISHED)
nodejs    12211    root 1018u  IPv4 151289731      0t0  TCP 10.101.42.209:36090->54.236.3.170:80 (ESTABLISHED)
nodejs    12211    root 1019u  IPv4 151314874      0t0  TCP 10.101.42.209:49176->54.236.3.172:80 (ESTABLISHED)
nodejs    12211    root 1020u  IPv4 151289768      0t0  TCP 10.101.42.209:45427->54.236.3.171:80 (ESTABLISHED)
nodejs    12211    root 1021u  IPv4 151289769      0t0  TCP 10.101.42.209:36094->54.236.3.170:80 (ESTABLISHED)
nodejs    12211    root 1022u  IPv4 151279903      0t0  TCP 10.101.42.209:43836->54.236.3.171:80 (ESTABLISHED)
nodejs    12211    root 1023u  IPv4 151281403      0t0  TCP 10.101.42.209:43930->54.236.3.172:80 (ESTABLISHED)
....

Notice the: 1023u (last line) —- that’s the 1024th file handle which is the default maximum.

Now, Look at the last column. That indicates which resource is open. You’ll probably see a number of lines all with the same resource name. Hopefully, that now tells you where to look in your code for the leak.

If you have multiple node processes, you can isolate it by using PID in 2nd column.

In my case above, I noticed that there were a bunch of very similar IP Addresses. They were all 54.236.3.###

So, I started doing iplookups on all the different 3rd party services I used… loggly, newrelic, pubnub… until I ultimately determined it was pubnub. Turned out we were creating a new socket each time we published an event instead of reusing.

Command Reference

Use this syntax to determine how many open handles a process has open…

To get a count of open files for a certain pid

I used this command to test the number of files that were opened after doing various events in my app.

lsof -i -n -P | grep "8465" | wc -l

root@ip-10-101-42-209:/var/www# lsof -i -n -P | grep "nodejs.*8465" | wc -l
28
root@ip-10-101-42-209:/var/www# lsof -i -n -P | grep "nodejs.*8465" | wc -l
31
root@ip-10-101-42-209:/var/www# lsof -i -n -P | grep "nodejs.*8465" | wc -l
34

What is your process limit?

ulimit -a

The line you want will look like this: open files (-n) 1024

Friday
Jan032014

Setting created and modified user via SugarCRM Webservice API

A Callinize user recently reported that they were getting email notifications for each Call being created. They had assignment notifications turned on, so whenever created_by_user !== assigned_user_id, an email gets sent to the assigned user that looks like:

{The_API_User} has assigned a Call to {Assigned_To_User}.

Subject: 
Status: 
Start Date: 2014-01-03 11:44:35 UTC(+00:00)
Duration: 0h, 3m
Description: {description}

You may review this Call at:
https://your.domain.com/index.php?module=Calls&action=DetailView&record=b64a1b7b-94f5-4b5d-01fd-52c6a2a33041

So, i figured oh we’re not setting the created user… this should be an easy fix. Well, for me at least… it wasn’t. Special thanks to Jason Eggers of SugarOutfitters or I would have never figured this out…

Flags Required

In order to modify them, you must include these flags. And no, it’s not a typo… they’re named inconsistently.

    "set_created_by" => false,
    "update_modified_by" => false,

Fields Needed

Again, not a typo… they’re just inconsistent.

    "created_by" => $userCrmId,
    "modified_user_id" => $userCrmId,

Final Snippet

Putting it all together, here’s the snippet I use in my set_entry calls:

    // Sets: assigned, created, and modified user to $userCrmId
    "set_created_by" => false,
    "update_modified_by" => false,
    "assigned_user_id" => $userCrmId,
    "created_by" => $userCrmId,
    "modified_user_id" => $userCrmId,

    // NOT NEEDED, Kept because:
    // 1. In case future API makes them more consistent. (dont want to revisit)
    // 2. Doesn't break anything (Surprisingly!)
    // 3. Will help people find this posting searching on google ;-)
    "set_modified_by" => false,
    "update_created_by" => false,
    "created_by_id" => $userCrmId,
    "modified_by_id" => $userCrmId,
    "modified_by" => $userCrmId,
    "created_user_id" => $userCrmId,

Note: I tested this with Sugar v4_1 api via rest. I’ve heard it might have changed in the newer rest apis that Sugar 7 has. If anyone knows, please add a comment.

EDIT: After going through all this… It turns out there is a bug in 6.7.1 which was the real problem. I think in Sugar 6.5 and below assignment notifications are only triggered for records which are updated (not created) and the assigned_user_id changes. Please upvote this issue: http://www.sugarcrm.com/support/bugs.html?caseID=63675#issue_63675

Monday
Jul292013

SugarCRM, SQL query to determine Sales Cycle From New Account --> Customer

SQL query below runs against SugarCRM Audit tables to determine how long it took an Account record to become a “Customer”.  It looks at the audit tables to calculate when the field was changed.

 

SELECT accounts.name, accounts.industry, accounts_audit.before_value_string, accounts_audit.after_value_string, 
accounts.date_entered,accounts_audit.date_created AS “Became Customer”, TIMESTAMPDIFF(DAY, accounts.date_entered, accounts_audit.date_created) AS “How Long”
FROM accounts_audit
JOIN accounts ON accounts_audit.parent_id = accounts.id
JOIN accounts_cstm ON accounts.id = accounts_cstm.id_c
WHERE after_value_string = “Customer” AND accounts.deleted = “0”

SELECT accounts.name, accounts.industry, accounts_audit.before_value_string, accounts_audit.after_value_string, accounts.date_entered,accounts_audit.date_created AS “Became Customer”, TIMESTAMPDIFF(DAY, accounts.date_entered, accounts_audit.date_created) AS “How Long”FROM accounts_auditJOIN accounts ON accounts_audit.parent_id = accounts.idJOIN accounts_cstm ON accounts.id = accounts_cstm.id_cWHERE after_value_string = “Customer” AND accounts.deleted = “0”

 

 

Wednesday
Jul242013

Creating an "API Only" User in SugarCRM

There are lots of reasons for API Only users, perhaps you are integrating with a marketing automation system, you developed a custom portal, or have a sugarcrm telephony integration (CTI) like Callinize.  In the past I had just used the admin account but this wasn’t ideal as if the credentials got out that user has ability to login.

The other thing I wanted was the ability to know when records were created by the system.  By creating a seperate user for your job the records would show up as created by “Your Integration User” instead of some other generic user. 

But, if you’re on a paid version of sugar, you might not want to fork over another $30+ a month.  The good news is there is a solution. CE users don’t care about this since they’re not paying per user.  Sugar Enterprise users by default have the “Create Portal User” option…. But what about us Sugar Pro customers?  Nothing seems obvious at first.

Fortunately, you can add a line to config_override.php and then you too will have the “Create Portal API User” option.

$sugar_config['enable_web_services_user_creation'] = true;

 

Now, when you go Admin -> User Management and mouse over the Users module tab there is an option for creating an “Portal API User” just like the Sugar Enterprise customers have.

 

Wednesday
Jun262013

SugarCRM Rest API Example: How to get all contacts for an account

This relies on the following PHP Wrapper Class: http://github.com/asakusuma/SugarCRM-REST-API-Wrapper-Class/

For a slightly better formatted answer see my Stackoverflow Post Here.

/**
 * returns an array of contacts that are related to the accountId passed as a param.
 * The array returned will be an array of associative arrays.
 * @param $accountId
 * @param array $contactSelectFields optional sets the different items to return, default includes id, email1, name, title, phone_work, and description
 * @return array
 *
 */
public function getAllContactsAtOrganization( $accountId, $contactSelectFields=array("id", "email1", "name", "title", "phone_work", "description")) {

    $sugar = new Sugar_REST( SUGAR_REST_URL, SUGAR_USER_NAME, SUGAR_PASSWORD);

    $fields = array( "Accounts" => array("id", "name"),
        "Contacts" =>  $contactSelectFields);
    $options = array(
        'where' => "accounts.id='$accountId'"
    );
    $apiResult = $sugar->get_with_related("Accounts", $fields, $options );


    $contacts = array();
    foreach( $apiResult['relationship_list'][0]['link_list'][0]['records'] as $almostContact) {
        $curr = array();
        foreach($contactSelectFields as $key) {
            $curr[$key] = $almostContact['link_value'][$key]['value'];
        }
        $contacts[] = $curr;
    }

    //print_r($contacts);

    return $contacts;
}

Sample Return

Array
(
    [0] => Array
        (
            [id] => 47e1376c-3029-fc42-5ae2-51aeead1041b
            [email1] => johndoe@gmail.com
            [name] => Blake Robertson
            [title] => CTO
            [phone_work] => 8881112222 
            [description] => Opinionated developer that hates SugarCRM's REST API with a passion!
        )

    [1] => Array
        (
            [id] => 4c8e3fcf-8e69-ed7d-e239-51a8efa4f530
            [email1] => csmith@mailinator.com
            [name] => Carolyn Smith
            [title] => Director of Something
            [phone_work] => 832-211-2222
            [description] => She's a smooth operator...
        )
)

For Reference Purposes

Here’s the “rest-data” (nicely formatted)

Used print_r of the php array

Array
(
    [session] => 9j7fm4268l0aqm25kvf9v567t3
    [module_name] => Accounts
    [query] => accounts.id='e583715b-7168-5d61-5fb1-513510b39705'
    [order_by] => 
    [offset] => 0
    [select_fields] => Array
        (
            [0] => id
            [1] => name
        )

    [link_name_to_fields_array] => Array
        (
            [0] => Array
                (
                    [name] => contacts
                    [value] => Array
                        (
                            [0] => id
                            [1] => email1
                            [2] => name
                            [3] => title
                            [4] => phone_work
                            [5] => description
                        )

                )

        )

    [max_results] => 20
    [deleted] => FALSE
)

Post Body

method=getentrylist&inputtype=JSON&responsetype=JSON&restdata={“session”:”iov5a257lk5acsg9l3ll6kuej3”,”modulename”:”Accounts”,”query”:”accounts.id=’e583715b-7168-5d61-5fb1-513510b39705’”,”orderby”:null,”offset”:0,”selectfields”:[“id”,”name”],”linknametofieldsarray”:[{“name”:”contacts”,”value”:[“id”,”email1”,”name”,”title”,”phonework”,”description”]}],”maxresults”:20,”deleted”:”FALSE”}method=logout&inputtype=JSON&responsetype=JSON&rest_data={“session”:”iov5a257lk5acsg9l3ll6kuej3”}

Thursday
Jun202013

Fulltext search engines for mediawiki: Solr vs Sphinx

I tried out both on our companies mediawiki instance.

Both provide much better search results then the default Mediawiki search.  Here are my thoughts…

Sphinx

Was slightly easier to setup.  Doesn’t include wiki tags in the search results.  Documentation was better.  Properly 

SOLR (lucene)

Prefered the search results.  I think it gives better weighting to title matches.  In my case, solr was already installed on my box (used a bitnami stack).  Documentation wasn’t as good.  My contractor had to email developer for a couple things.  Other pro is there are more tutorials for getting other information into SOLR then their are sphinx.  In my case I’m going to be indexing my mediawiki instance plus my knowledgebase so this made it a no brainer for me.

I plan on figuring out how to strip the mediawiki tags out and get it to highlight the keyword matches.  I’ll post back when I do. 

Here’s some screenshots:

 

 

Thursday
Jun132013

Generating an avatar for customers with a splash of utility

It’s kind of a pet peave of mine when I see a bunch of empty contact photos. So, when we started rolling out a support desk program (we’re using FreshDesk), the last thing I wanted every day was to have a bunch of blank contact photos staring back at me.

I recently wrote a script which will generate icons for customers who haven’t set a profile picture. The result I think has some nice utility to it as well…

For each customer, I generate an icon consisting of their initials and and then a rectangular bar of color. The color of their initials is unique to them… and the bar of color is the same for each customer in an organization.  This allows you to glance at the list of tickets and more easily recognize the contacts / companies.  

To generate the “random” color, I simply did an MD5 hash of their Name or Company Name.  Then, took the first 6 digits and made that hex value my color.  Pretty easy.

I wrote a script and API wrapper for FreshDesk which can iterate through all the contacts for your organization and update them all in one step!  That code is available on github here: https://github.com/blak3r/freshdesk-solutions.  

In case you’re wondering, why didn’t I just use gravatar?  Only about 5% of our customers have their work email addresses (which we correspond with) associated with a gravatar profile.  So, this is the next best alternative.

There’s a lot of other cool things you could do with something like this… You could have maybe special colors indicating support priority levels… add little stars or flags maybe… Please post back if you come up with a better design.  I ended up using Future MD Cn Bold.  

 

 

Tuesday
Feb192013

SugarCRM - Update Calculated Fields Nightly

SugarCRM has some powerful calculated fields called Sugar Logic. One limitation though is they are only calculated when a field is saved. I wanted a simple way to display the number of days since X occurred.

So, I created a very simple script which can be run from the Scheduler every night. It basically just iterates through all the records for a user configurable module type and forces the calculation to update. Best of all it does this without changing the modified date, modified by or creating a tracker entry.

This script could also be used one time to just initially seed all the calculated values for a module. The suggestion from the admin guide is to do a mass update but that’ll cause modified dates, etc to change.

The code is hosted on github here: https://github.com/blak3r/sugarcrm-update-calculated-fields

Potential Applications:

  1. Days since lead was last contacted.
  2. Days since order added to system.
  3. Days since a support case was entered.
  4. (There’s lots)

My original application was to keep track of how long it was taking our company to ship orders once we had all the information from a customer.

  1. I created a workflow rule such that when orderstage == Production Ready, I would set the date field “productionready_date”.
  2. I then created a calculated field called “dayssinceproduction_ready” and set the formula to something like abs(daysSince($production_ready_date))
  3. Installed the script on my github site and created a scheduler job to run at 3am every night.

We now use the field as a dashlet on our order status tab and it’s used in some reports to be able to look back over the past year and see how well the production team is doing at getting orders out and to be able to do analysis on the orders that took a long time to identify ways to speed them up.

Alternate Strategies

An alternative to using custom fields would be to create a nightly task which just updates a fields. I prefer this approach for the following reasons…

  1. I can add new calculated fields using Studio and not have to update code at all on the server. You can also have multiple calculated fields per module.
  2. Since Sugar Logic is also applied in Javascript fields, if a date field is changed manually (instead of via a workflow rule), the calculated field updates in realtime while in the edit view and will be accurate on the detail view as well. Otherwise you’d need to create a on save logic hook to do that if nightly synchronization wasn’t enough.

Other alternative is to not calculate the days and just have a date field and design reports to only show records that are within a certain time range. This was the initial approach I used to take, I find that this makes creating reports a LOT harder. Also, if you want to calculate something like the average time an order was fulfilled you can’t do that with Sugar’s Reporting at this time. Plus it’s just easier in my opinion to look at an integer such as 14 vs. 2013-02-18. It also simplifies custom formatting of rows in my dashlets. It’s fairly easy to write some jquery code to highlight based on a number… having to parse dates and calculate them would be slower and more of a pain.

Tuesday
Nov272012

SnagIt - Why I Hate It and Yet Can't Live Without It...

I have a love hate relationship with Techsmith's SnagIt. The app does so many things very well... but there are few things which really frustrate me on a daily basis. I go searching for alternatives fairly often as a result of them. Here's my list of features which are really missing in SnagIt.

At the time of writing, The latest versions are: SnagIt 11.1 on Windows and SnagIt 2.0 on Mac.

1. Facilitate attaching images to Gmail/Web Apps

Webmail is great... but until browsers can fully access the clipboard, pasting images into a gmail window isn't going to be possible. The solution to this limitation that most webapps (gmail included) have embraced is allowing images to be "drag and dropped."

On SnagIt for windows, when you drag and drop an image from the catalog bar (the bar that shows up under the image editor), it attaches the image in the proprietary .SNAG format. Solution: create an event handler for this such that when you drag the image from the catalog it automatically saves it as a .png file in the catalog and use that copy as the image to drop.

On SnagIt for Mac, you cannot drag images out of the catalog bar at all. So, this needs to be added.

The fastest way to attach images I've found is to first do "Save as" and save as a png. Once you do that you can then drag it over and it'll attach as png. But, the additional save as step slows things down. Especially, you were trying to markup 5 images for an email. That extra save as step really adds up. So, the simple solution to this is add an event handler to detect dragging, automatically save the image as a PNG or JPG and then have that copy get "dropped" in the application instead of the .SNAG version.

2. Ability to resize everything using keyboard shortcut.

If you've ever used Balsamiq Mockups, you'll know what i'm talking about. In that app CTRL+[ makes things smaller and CTRL+] makes things bigger. For example, if you have text selected, it should make the text bigger. If you have an arrow selected, it should make the width of the arrow thicker. If you have a rectangle selected, it should make the stroke around it thicker. If you have a stamp selected, it should scale it up and down.

Having to hover over things to reveal the context menu is incredibly slow.

In SnagIt for windows, it is possible to create shortcuts for resizing the text. But, it's not possible to assign a shortcut to increase the width of an arrow unfortunately.

3. Scale Font To Fit.

When I draw a rectangle, I generally know that's the size I want it to be. I want the text I compose to be as large as possible but still fit in the area I defined. So, the default font size should be AUTO and it should resize the font to make this happen for me automagically.

The Mac version is slightly better. When you add more text then what would fit in the text area it automatically expands the shape. But, I'd prefer it to just scale the font instead because as I said earlier... Generally when I draw the shape, that's the size I want it to fit in.

4. Better Default Styles -or- Quicker Editing of Styles

If you love "Comic Sans" then you're the type of person that would like the default styles. Bad Fonts, Bad Colors, and Unprofessional shapes. This feedback applies more to the windows version. The Mac team did a much better good job of defining the styles.

The workaround to this that most people would suggest is to "Create a custom style". Well, no custom styles don't work (see below). But, if you're going to insist on it then at the very least they need to be improved in the following ways:

  1. Custom Styles need to be at the top of the list... not at the bottom where I have to scroll down every time for. Alternatively, make it possible to just delete the default quick styles.
  2. It should remember the last custom style used between sessions and default to this rather then the ugly default style.
  3. The preview for a custom styles don't look right. They don't show my font half the time or it's hard to tell how big the arrow widths are.

Better yet! Just make it easier to change the color, size, etc. No matter how organized you are about creating quick styles, there is no substitute for being able to customize the attributes on the fly. The colors I choose are directly related to the image background / style I'm marking up... so you you simply can't create styles for every occasion. Back in SnagIt 8, life was good. The original interface was so fast and intuitive. Once they made the switch to the ribbon interface in SnagIt 9, Things got hidden in menu's which appear on hover. God forbid you want to change the font size, font, color, and say the width. That's 3 separate hover steps. I still have snagit8 installed and prefer it's editor over the ribbon style anyday!

I like the Mac Version much better because it style has a property panel (similar to how SnagIt 8 for Windows did). But, the Mac Design team made some poor decisions for this panel. For example, If i'm editing a text rectangle. They give you options to quickly edit the Line Style and Width. But, changing the Font size, type and color are buried under a separate panel. Who edits the rectangle width more often then font size? No one! So, these options should be given priority.

Make it Easier to Combine Multiple Captures.

Often I'll take several captures and want to combine them into a single display. When you try and move the second capture into the first, it should automatically expand the canvas area.

Some more nit picky feature requests...

The features above would satisfy me and I feel all are completely actionable requests. But, here's a complete feature request list.

  1. On Mac, the padding between rectangle edge and the text is too high. Hard to fit everything in.
  2. Mac version, doesn't have features like Trim, Cutout, Spotlight and magnify at all.
  3. Windows Version, doesn't have the reflection option.
  4. On Windows, would love to be able to do spotlight & Magnify on multiple areas.
  5. Both Versions, would be nice to be able to create custom stamp library from multiple sources.
  6. On Mac, no option to assign a keyboard shortcut to Increase/Decrease Font size at all.
  7. Create styles which are proportional to image resolution. The styles are by default sized appropriately for screen captures at screen resolutions. If you ever try to markup an image from a digital camera, you'll need to increase the widths / font sizes. Make them be a percentage or scale appropriately based on source image resolution.

No better option!?

So, I've outlined a few of my biggest grips with SnagIt and all of them have to do with the built in editor. There is probably a better editor out there somewhere. But, I keep coming back to SnagIt for the following reasons. If anyone knows of a better editor that I could tie into SnagIt for editing please chime in.

  1. I love how I can capture things and not have to save them. I frequently take a screen grab and then do nothing with it. I save it for later when I have time to do something with it. So, it getting captured in Snagit's catalog is the unique feature which keeps me from switching.
  2. The "All in One" Capture option is incredible.
  3. SnagIt is reasonably priced ($50). I use Snagit on a lot of computers so putting an Adobe Creative Suite on all of them is just cost prohibitive.
  4. SnagIt is lightweight. The time to go from capture to editing is well under 2 seconds (if you have a good computer at least)... firing up an Adobe app just takes too long.
  5. For the most part... everything I need is in SnagIt and it's faster and easier then more heavyweight apps.

I'd really like a better solution and would love for people to chime in on the comments and let me know what they use and if there is a better solution out there. Thanks!

Update: 1/11/2013

I evaluate some other applications.

  1. Most promising to me is Clarify. It doesn't have a catalog per se but you can capture lots of images into a single book. It's intended for making a step by step instruction list. So, it allows you to put blocks of text between your screen grabs. It's a pretty good idea. I really like how they have styles. It's very easy to modify the colors widths etc.

  2. Skitch - seems to be the trendy app people are using. They have very limited options. You get one font and a small color palette. But the one font looks nice, an the colors all go together.