Blog

Building Kick SaaS Installation Profiles

Posted on: 2016-09-16 20:36:21

So back in April of this year, I gave a talk at Texas Camp entitled Building Kick SaaS Installation Profiles.

The slides are over there, but I'm uploading a local copy of them here—just in case!

Note: no audio is available, but the slides might be pretty handy!

How to monitor any text file import

Posted on: 2016-07-08 15:18:51

So as I was sitting here just now waiting for an 80+ MB import to process over an SSH connection, I began wondering to myself if there was some way to monitor the progress of an import. Percentage—or even throughput—of the file being sent over the pipe.

Turns out that there is a way!

Enter the command pv.

pv allows a user to see the progress of data through a pipeline, by giving information such as time elapsed, percentage completed (with progress bar), current throughput rate, total data transferred, and ETA.

Uh. Yes, please.

If you're running GNU/Linux, you probably already have this installed. On a Mac it's available via Homebrew.

brew install pv

Once installed, you use it to send your text files to commands that are expecting things through stdin like so:

$ pv some-db.sql | mysql -uroot 8f00b204e9800998ecf8427e
83.2MiB 0:00:25 [3.28MiB/s] [==================================================================>] 100%

Never worry about where your imports are again!

Taking a look at output buffering in PHP

Posted on: 2016-06-26 23:19:43

Output buffering in PHP isn't new. It was introduced in PHP 4 and for the most part, it's one of those things that unless you're not using a CMS or a framework, you'll never really need to mess with it much.

However, today I was playing around with doing client-side redirections in JavaScript and wanted to see first-hand how they worked. Modern browsers begin parsing the HTML before it is completely delivered and once the <script> tag is parsed, it is executed immediately.

So if we have a <script> tag just inside of our <head>, then that script tag will get executed as soon as it is done, regardless of the rest of the content in the delivered HTML file.

Anyway, to play around with this, I used this little bit of code:

<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01//EN\">
<html>
  <head>
    <title>Redirect test</title>
  </head>
  <body>
    <script>location.href=\"https://google.com/\"</script>
    <?php
      flush();
      sleep(10);
    ?>
  </body>
</html>

However, it did not work as intended.

I tried several different things trying to get it to work, but it wasn't until I added echo str_pad('', 64 * 1024).\"\\n\"; before the sleep that it actually worked:

<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01//EN\">
<html>
  <head>
    <title></title>
  </head>
  <body>
    <script>location.href=\"https://google.com/\"</script>
    <?php
      echo str_pad('', 64 * 1024).\"\\n\";
      sleep(10);
    ?>
  </body>
</html>

So, why did it need another 64k worth of stuff in order to have the intended effect? Well, the answer, of course, is buffering!

Yet another buffer!

I'm using Acquia DevDesktop for local development. It uses PHP's standard buffer size of 4096. No big deal there. I trying backing down the str_pad() to that and it still wouldn't send the data.

Ultimately, what I discovered was that FastCGI has a FcgidOutputBufferSize directive and it's value is actually set to—get this—65536.

So ultimately what was happening was that PHP's buffer was getting sent, but only to the handler (that'd be FastCGI) and then FastCGI was buffering the output itself.

It's also worth noting that using mod_deflate has it's own internal buffer (set by the DeflateBufferSize directive). I had it disabled for my test—and I'm not 100% sure of the impact that it might have.

One other quick note about flush(): As far as I can tell, each buffer will capture data until it gets full. Once it gets full, it will send it's data on to the next buffer. Say for a moment you have a sleep() call in your code that suspends execution. How long will it take for a the first byte to get delivered?

Buffer	6K	11K	16K
PHP, 4K	2K	3K	0
FastCGI, 8K	4K	8K	8K
Sent to browser	0	0	8K

In other words, the first buffer (PHP's) fills up and then sends it's data over to the next buffer (FastCGI's). FastCGI won't send it's buffer to the user until PHP delivers enough content to fill up it's own buffer so that it needs to empty it's buffer to start filling it up again.

It's something to keep in mind. And again, if Apache or something else is doing extra caching, you'll need to ensure you either bypass those buffers or send enough data so that buffers get flushed.

Initial impression of the Hamshield

Posted on: 2016-05-23 09:12:32

Background and Intro

The Hamshield was a Kickstarter that ended sometime last summer. The original expected delivery date was October of 2015, but due to issues with manufacturing and a problem with their amplification circuits, it was delayed. I won't go into the details here, but you can read it on their Kickstarter page:

They started shipping about 3-4 weeks ago.

I got my Hamshield about 2 weeks ago. I was wanting to do more with it once I got it, but we're in the process of moving (and since it required some assembly & soldering) it had to wait until I could bust out the old soldering iron before I could do anything with it. Anyway, here are my thoughts and initial impression.

Assembly

Soldering on the headers wasn't bad at all. Once I got started, it took about 5 minutes to work through them. The GND pins by VIN didn't seem to want to solder very well at all. But everything else went very quickly.

The GND pins needed more love

The SMA connector was a bit much for my little 15 watt iron. Of course, all I had was a fine-tip on my iron, so a bigger tip would work much better on the SMA. My suggestion is to get the two pins on the back first, since they are smaller and tinned and tacked down much more easily than the pins on the tip of the board. I've got a 20W iron somewhere and I'll probably resolder this once I find it and put a bigger tip on it.

The only other "gripe" I had about the SMA jack (and this is really just due to my lack of a good tool) is that there are components really close to the jack. I'm talking about C19 and R9, specifically. It was tough to get the iron good and close to the pins. Take a look:

SMA pins did not solder well and there were components very close to them

The only other real thing that bothered me was the proximity of the headers near the USB port on the Uno:

Close-up of the header pins and the USB jack on the Uno

I stuck a piece of electrical tape on there just to be safe. It looks a little close for comfort.

Oh, and one other note. My shield didn't come with pass-thru headers... like most shield would have so that you could stack them. Not sure if that was a mistake or not, but it would be nice.

Using the Examples

The folks over at Enhanced Radio Devices provide an Arduino library on Github so that you can use and build stuff with your new shield.

Out of the box, I couldn't get the libraries to compile. I've submitted a pull-request to fix the issue, which appears to just be a pre-processor define that uses a hyphen in the name.

Once that was fixed, I was able to get the FMBeacon example loaded and going on the device. The audio was a bit anemic, but it does work:

However, many of the other examples did not compile, JustTransmit does not work and some of the other ones do not work on Uno due to memory constraints.

There is also a reference in some of the applications to a PCM.h. There doesn't appear to be any documentation of what that is, specifically, but there are some PCM libraries for Arduino. So perhaps those would work.

Review

Well, I'm still excited about using the Hamshield. Though, it still seems like there is much to be cleaned up in the Arduino library and example applications. It'd be nice to be able to test things a bit better and I'm regretting not getting some sort of case for it since a 5W transmitter within a couple of feet will cause the unit to reboot. (I was trying to test the RSSI readings and it would just reboot while I tried this using a Baofeng HT).

Personally, I'd recommend getting some SMA pigtails and using that instead of the supplied antenna, but it will work in a pinch. It's hard to handle this thing with the antenna fully extended.

That's all for now. Thanks Casey and the folks at ERD for their efforts—looking forward to using it in my first real project!

Importing a price in Drupal 8 for Drupal Commerce 2.x

Posted on: 2016-04-20 14:21:04

Just a quick note that if you're importing a Commerce price:

you must set the currency_code.
what was commerce_price in DC 1.x / Drupal 7 is now just price.
the number can be a decimal now.

'price/number': SomeFieldName 'price/currency_code': plugin: default_value default_value: 'USD'

Note that you can also set up a constant in the migration definition and use the constant value:

source:
  plugin: d6_ubercart_product_variation
  constants:
    currency: USD
...
process:
  'price/number': SomeFieldName
  'price/currency_code': 'constants/currency'

It is worth mentioning that this fails silently right now. That is, if you don't provide the currency code, the migration will continue but you just won't have a value.

Update 2017-11-20: Vimokkhadipa told me that price/amount in the example above should actually be price/number. Since I haven't done any migration work in a while I'm taking his word for it and noting the update. Thanks Vimokkhadipa!

You can chain callbacks in a Drupal 8 migration

Posted on: 2016-04-20 09:08:53

If you're wanting to do some advanced processing (e.g. lowercasing a word and then capitalizing the first letter) you can chain the two callbacks to get the desired effect:

process:
  name:
    -
      plugin: callback
      source: MyVar
      callable:
        - '\\Drupal\\Component\\Utility\\Unicode'
        - 'strtolower'
    -
      plugin: callback
      callable:
        - '\\Drupal\\Component\\Utility\\Unicode'
        - 'ucfirst'

The two transformations will take place one after the other. The first one will execute and pass in the value straight from the source (e.g. 'SOMEVAR'). The next one will get the result (e.g. 'somevar') and then the the final value will be ('Somevar').

Quickly importing new config entities from the command line using Drush

Posted on: 2016-04-18 13:58:03

Drush's cset command is a bit picky, but with a few command line options, we can get it to import a yaml file into our site's config:

drush -y --format=yaml cset config.entity.string '' - < config_file.yml

This will put the entirety of config_file.yml into the configuration entity config.entity.string.

The -y is required because drush will ask you if you want add a new entity (and of course you do!). The "trick" if it even is that, is the '' before the - to read from STDIN.

Field Tools as a replacement for Features

Posted on: 2016-04-04 12:28:56

One of my new favorite modules recently has been Field Tools. What I love about it is that it presents you with the ability to export all of the fields on a particular bundle (or even just a single field).

Then, you can just use field_create_field or field_create_instance in a little loop.

Take for example (in a module's .install file):

<code lang="php"> function mymodule_install() { $fields = mymodule_get_some_fields();

foreach ($fields as $name => $field) { field_create_field($field['field']); field_create_instance($field['instance']); } }

Then, inside of mymodule_get_some_fields() you just put the output of Field Tool's export:

function mymodule_get_some_fields() {
  $fields = array();
  // Start export.
  $fields['field_name'] = array(
    'field' => array(...),
    'instance' => array(...),
  );
  $fields['another_field_name'] = array(
    'field' => array(...),
    'instance' => array(...),
  );
  // End export.
  return $fields;
}

See, the magic is that the $fields export includes two sub-arrays: 'field' and 'instance'. That's what then powers the ability to just run it through the foreach above and it create your bases and your instances.

A couple of caveats:

You should probably check to see if the field exists before creating it. Otherwise, it'll throw an Exception.
It dumps everything in the exports, and so you might need to clean them up a bit by checking to ensure that you're not adding them to extra bundles or something like that.

But for most cases, this is way simpler than using Features and having to bring it in as a dependency.

A quick note about Drupal Commerce price updaters that use Feeds and Feeds Validation

Posted on: 2016-03-31 14:25:21

Perhaps this may save someone else some time:

If you're wanting to do price updating in Drupal Commerce using Feeds, you'll normally use a field that is guaranteed unique. The Product ID and Product SKU fields (provided by the Commerce Feeds module) give this to you out of the box, but if you're wanting to give a user the ability to match by something else: say, a catalog number or some vendor part number that is an actual field on the product entity itself, you might have already looked at Field Validation to do this.

The general idea goes like this:

Set up field validation so that the field you're wanting to use has the "Unique value" validator.
Now, in Feeds, that field will show as being able to be used as a unique field.

This works for fields that are used on a single entity bundle, but not for fields that are used on multiple entities.

The problem arises when it tries to read the configuration for the bundles out of the field_validation_rule table. It might work if you have it defined for every instance of the field on every bundle, but I haven't tried that.

Make your life easier by just using Feeds Tamper String2ID, and creating a view that does a lookup from whatever field you're wanting to match with to the Product ID. Then, let Commerce Feeds' ProductProcessor do the rest.

Modifying distribution Makefiles within your own project Makefile on Platform.sh

Posted on: 2016-02-24 09:48:42

As I've written about in the past, Platform.sh's Makefile build system is pretty sweet. One of the things that confuse me the most, however, is how to make a change to a distribution (such as adding a patch) without making things even more complicated.

Hat tip to Tavi Toporjinschi for writing the original example this is based off of.

Let's say for a moment that you want to add a patch to a module included in a distribution. If you're just starting out, you could actually just override the module and add it to sites/default/modules by including it within your existing file like so:

projects[commerce_paypal][version] = \"2.3\"
projects[commerce_paypal][patch][2082691] = \"https://www.drupal.org/files/issues/2082691-24-support_paypal_adaptive_payments_-chained_0.patch\"

However, if you're wanting to change a module that you've already been using, you'll want to modify the distribution directly from within your Makefile.

To do this, start by grabbing a copy of the distribution's git repository (and make sure you're working from the same version that you're using in your Makefile!). Here's instructions for Commerce Kickstart. Once you've done that, open up drupal-org.make which is the main Makefile for Commerce Kickstart.

In this particular instance, what I'd like to do is use the -rc4 version of Commerce Message. So I'll go ahead and make that change by finding the line that says:

projects[commerce_message][version] = 1.0-rc3

It will now read:

projects[commerce_message][version] = 1.0-rc4

Now that I've got my change and I've tested it (which I'm skipping for now), it's time to add it back into my project. To do this, we'll grab a diff of the change:

commerce_kickstart$ git diff > commerce_message_to_rc4.patch

The file commerce_message_to_rc4.patch will look something like this:

diff --git a/drupal-org.make b/drupal-org.make
index 45fef75..b497a53 100644
--- a/drupal-org.make
+++ b/drupal-org.make
@@ -48,7 +48,7 @@ projects[commerce_moneybookers][patch][] = \"http://drupal.org/files/commerce_mon
 projects[commerce_paypal][version] = 2.3
 projects[commerce_paypal][patch][2458721] = \"https://www.drupal.org/files/issues/commerce_paypal-addressfield-default-values-2458721-1.patch\"
 projects[commerce_backoffice][version] = 1.5
-projects[commerce_message][version] = 1.0-rc3
+projects[commerce_message][version] = 1.0-rc4
 projects[commerce_search_api][version] = 1.4
 projects[commerce_add_to_cart_confirmation][version] = 1.0-rc2
 projects[commerce_kiala][version] = 1.0-rc1

To add this to my project, we'll create a patches/ directory within the project (if it doesn't already exist) and copy that patch into the directory.

commerce_kickstart$ mkdir ~/Platform.sh/myproject/repository/patches commerce_kickstart$ cp commerce_message_to_rc4.patch ~/Platform.sh/myproject/repository/patches/

Now, we just need to patch the distribution within the Makefile. How do we do that? With a patch, of course! You probably have a line like this in your makefile:

projects[commerce_kickstart][type] = core

This downloads the "core" version of Commerce Kickstart. This is the one that you get when you download directly from Commerce Kickstart's module page or when you download commerce_kickstart-7.x-2.33-core.tar.gz from the releases page.

This will then be replaced with the following:

projects[drupal][type] = core
projects[commerce_kickstart][type] = profile

; Use dev git version with patch.
projects[commerce_kickstart][download][type] = git
projects[commerce_kickstart][download][branch] = 7.x-2.x
projects[commerce_kickstart][patch][] = \"patches/commerce_message_to_rc4.patch\"

And... breaking it down:

projects[drupal][type] = core signals that we're using Drupal core as the base install. Then, we're installing commerce_kickstart as a profile on top of that (projects[commerce_kickstart][type] = profile). When installing the profile, we specify git as the download type and the branch (7.x-2.x) that we are working on so that we get the correct version of the code that we worked on up above (which include's its Makefile). Finally, projects[commerce_kickstart][patch][] = \"patches/commerce_message_to_rc4.patch\" patches the distribution so that it downloads the -rc4 version of Commerce Message.

Now, build the project, if everything is correct, as the build goes by, you'll see the correct module being downloaded:

 >> commerce_authnet-7.x-1.1 downloaded.                             [ok]
 >> commerce_message-7.x-1.0-rc4 downloaded.                         [ok]
 >> commerce_physical cloned from                                    [ok]
http://git.drupal.org/project/commerce_physical.git.

And there we go. Now, push it up to your repository, and you're all set!