Improving Drupal 8's API-first: JSON API and OAuth2?

Among the most important initiatives for Drupal 8's future is the "API-first initiative", whose goal is to improve Drupal's web services capabilities for building decoupled applications. In my previous blog posts, I evaluated the current state of, outlined a vision for, and mapped a path forward for Drupal's web services solutions.

In the five months since my last update, web services in Drupal have moved forward substantially in functionality and ease of use. This blog post discusses the biggest improvements.

An overview of the key building blocks to create web services in Drupal. Out of the box, Drupal core can expose raw JSON structures reflecting its internal storage, or it can expose them in HAL. Note: Waterwheel has an optional dependency on JSON API if JSON API methods are invoked through Waterwheel.js.

Core REST improvements

With the release of Drupal 8.2, the core REST API now supports important requirements for decoupled applications. First, you can now retrieve configuration entities (such as blocks and menus) as REST resources. You can also conduct user registration through the REST API. The developer experience of core REST has improved as well, with simplified REST configuration and better response messages and status codes for requests causing errors.

Moreover, you can now access Drupal content from decoupled applications and sites hosted on other domains thanks to opt-in cross-origin resource sharing (CORS) support. This is particularly useful if you have a JavaScript application hosted directly on AWS, for instance, while your Drupal repository is hosted on a separate platform. Thanks to all this progress, you can now build more feature-rich decoupled applications with Drupal.

All of these improvements are thanks to the hard work of Wim Leers (Acquia), Ted Bowman (Acquia), Daniel Wehner (Chapter Three), Clemens Tolboom, José Manuel Rodríguez Vélez, Klaus Purer, James Gilliland (APQC), and Gabe Sullice (Aten Design Group).

JSON API as an emerging standard

JSON API is a specification for REST APIs in JSON which offers several compelling advantages over the current core REST API. First, JSON API provides a standard way to query not only single entities but also all relationships tied to that entity and to perform query operations through query string parameters (for example, listing all tags associated with an article). Second, JSON API allows you to fetch lists of content entities (including filtering, sorting and paging), whereas the only options for this currently available in core are to issue multiple requests, which is undesirable for performance, or to query Drupal views, which require additional work to create.

Moreover, JSON API is increasingly common in the JavaScript community due to its adoption by developers creating REST APIs in JSON and by members of both the Ruby on Rails and Ember communities. In short, the momentum outside of Drupal currently favors JSON API over HAL. It's my belief that JSON API should become an experimental core module, and it may one day potentially even supersede HAL, Views-based REST endpoints, and more. Though Views REST exports will always be valuable for special needs, JSON API is suitable for more common tasks due to query operations needing no additional configuration. All this being said, we should discuss and evaluate the implications of prioritizing JSON API.

Thanks to the efforts of Mateu Aguiló Bosch (Lullabot) and Gabe Sullice (Aten Design Group), the JSON API module in Drupal 8 is quickly approaching a level of stability that could put it under consideration as a core experimental module.

OAuth2 bearer token authentication

One of the issues facing many decoupled applications today is the lack of a robust authentication mechanism when working with Drupal's REST API. Currently, core REST only has two options available out of the box, namely cookie-based authentication, which is unhelpful for decoupled applications on domains other than the Drupal site, and basic authentication, which is less secure than other mechanisms.

Due to its wide acceptance, OAuth2 seems a logical next step for Drupal sites that need to authenticate requests. Because it is more secure than what is available in core REST's basic authentication, OAuth2 would help developers build more secure decoupled Drupal architectures and allow us to deprecate the other less secure approaches.

It would make sense to see an OAuth2 solution such as Simple OAuth, the work of Mateu Aguiló Bosch (Lullabot), as an experimental core module, so that we can make REST APIs in Drupal more secure.

Waterwheel, Drupal's SDK ecosystem

The API-first initiative's work goes beyond making improvements on the Drupal back end. Acquia released Waterwheel.js 1.0 as open-source, the first iteration of a helper SDK for JavaScript developers. The Waterwheel.js SDK helps JavaScript developers perform requests against Drupal without requiring extensive knowledge of how Drupal's REST API functions. For example, the Waterwheel module allows you to benefit from resource discovery, which makes your JavaScript application aware of Drupal's data model. Waterwheel.js also integrates easily with the JSON API contributed module.

Thanks to Kyle Browning (Acquia), the Waterwheel.swift SDK allows developers of applications for Apple devices such as iPads, iPhones, Apple Watches, and Apple TVs to more quickly build their Drupal-powered applications. To learn more about the Waterwheel ecosystem, check the blog posts by Preston So (Acquia).

Conclusion

Thanks to the hard work of the many contributors involved, the API-first initiative is progressing with great momentum. In this post, we ended with the following conclusions:

  • The JSON API and Simple OAuth modules could be candidates for inclusion in Drupal 8 core — this is something we should discuss and evaluate.
  • We should discuss where support for HAL and JSON API stops and starts, because it helps us focus our time and effort.

If you are building decoupled applications with Drupal 8, we would benefit significantly from your impressions of the core REST API and JSON API implementations now available. What features are missing? What queries are difficult or need work? How can we make the APIs more suited to your requirements? Use the comments section on this blog post to tell us more about the work you are doing so we can learn from your experiences.

Of course, if you can help accelerate the work of the API-first initiative by contributing code, reviewing and testing patches, or merely by participating in the discussions within the issues, you will not only be helping improve Drupal itself; you'll be helping improve the experience of all developers who wish to use Drupal as their ideal API-first back end.

Special thanks to Preston So for contributions to this blog post and to Wim Leers, Angie Byron, Ted Bowman and Chris Hamper for their feedback during the writing process.

Back to school

Last week I presented at the University of Antwerp, my alma mater. I was selected to be the 2016/2017 ambassador of the alumni and was asked to talk about my career and work. Presentations like this are a bit surreal because I still feel like I have a lot to learn and accomplish. Deep down I'll always be searching for something more. I want my life and career to be meaningful and creative, and full of laughter and friends. This presentation was very special as it was attended by my parents, friends from high school and college, professors whose classes I attended 20 years ago and the university's rector or chancellor, Herman Van Goethem. It was great to laugh and catch-up with old friends and family, and it felt meaningful to share some of my lessons learned to a group of young students.

Antwerp university presentation
The university's rector or chancellor, Herman Van Goethem, introducing me.
Antwerp university presentation
My parents sitting on the front row.
Antwerp university presentation
Antwerp university presentation
Antwerp university presentation
Antwerp university presentation
Me with some of my friends from high school that I hadn't seen in 20 years!

Drupal 8 turns one!

Tomorrow is the one year anniversary of Drupal 8. On this day last year we celebrated the release of Drupal 8 with over 200 parties around the world. It's a project we worked on for almost five years, bringing the work of more than 3,000 contributors together to make Drupal more flexible, innovative, scalable, and easier to use.

To celebrate tomorrow's release-versary, I wanted to look back at a few of the amazing Drupal 8 projects that have launched in the past year.

1. NBA.com

The NBA is one of the largest professional sports leagues in the United States and Canada. Millions of fans around the globe rely on the NBA's Drupal 8 website to livestream games, read stats and standings, and stay up to date on their favorite team. Drupal 8 will bring you courtside, no matter who you're rooting for.

2. Nasdaq

Nasdaq Corporate Solutions has selected Drupal 8 as the basis for its next generation Investor Relations Website Platform. IR websites are where public companies share their most sensitive and critical news and information with their shareholders, institutional investors, the media and analysts. With Drupal 8, Nasdaq Corporate Solutions will be providing companies with the most engaging, secure, and innovative IR websites to date.

3.Hubert Burda Media

For more than 100 years, Hubert Burda Media has been Germany's premier media company. Burda is using Drupal 8 to expand their traditional business of print publishing to reach more than 52 million readers online. Burda didn't stop there, the media company also open sourced Thunder, a distribution for professional publishers built on Drupal 8.

4. Jurassic World

Drupal 8 propels a wide variety of sites, some of Jurassic proportion. Following the release of the blockbuster film, Jurassic World built its digital park on Drupal 8. Jurassic World offers fans games, video, community forums, and even interactive profiles all of the epic dinosaurs found on Isla Nublar.

5. WWF

The World Wide Fund for Nature has been a leading conservation organization since its founding in 1961. WWF's mission is to protect our planet and Drupal 8 is on their team. WWF UK uses Drupal 8 to engage the community, enabling users to adopt, donate and join online. From pole to pole, Drupal 8 and WWF are making an impact.

6. YMCA Greater Twin Cities

The YMCA is one the leading non-profit organizations for youth development, healthy living, and social responsibility. The YMCA serves more than 45 million people in 119 countries. The team at YMCA Greater Twin Cities turned to Drupal 8 to build OpenY, a platform that allows YMCA members to check in, set fitness goals, and book classes. They even hooked up Drupal to workout machines and wearables like Fitbit, which enables visitors to track their workouts from a Drupal 8 powered mobile app. The team at Greater Twin Cities also took advantage of Drupal 8's built-in multilingual capabilities so that other YMCAs around the world can participate. The YMCA has set a new personal record, and is a great example of what is possible with Drupal 8.

7. Jack Daniels

The one year anniversary of Drupal 8 is cause for celebration, so why not raise a glass? You might try Jack Daniels and their Drupal 8 website. Jack Daniels has been making whiskey for 150 years and you can get your fill with Drupal 8.

8. Al Jazeera Media Network

Al Jazeera is the largest news organization focused on the Middle East, and broadcasts news and current affairs 24 hours a day, 7 days a week. Al Jazeera required a platform that could unify several different content streams and support a complicated editorial workflow, allowing network wide collaboration and search. Drupal 8 allowed Al Jazeera to do that and then some. Content creators can now easily deliver critical news to their readers in real time.

9. Alabama.gov

From Boston to LA and even Australia, Drupal is supporting the digital needs of governments around the globe. Alabama is leading the way with Drupal 8. Alabama.gov puts its citizens first, and demonstrates how open source can change the way the public sector engages online.

10. Box

Box has been a leader in the technology industry since its founding in 2005. Box takes advantage of Drupal 8 and the improved features made available right out-of-the-box. Bad puns aside, companies like Box are using Drupal 8's new features and improved user interface to build the best digital experiences yet.

11. Habitat for Humanity

The historic nonprofit Habitat for Humanity doesn't just build houses for those in need; they build habitat.org on Drupal 8. Habitat for Humanity provides affordable housing for communities in over 70 countries around the world. You can discover their impact through the "Where we Build” interactive map, donate, and volunteer all on their Drupal 8 site.

12. Obermeyer

Obermeyer and Drupal 8 will take you into new territory. The ski wear company offers seamless end to end commerce integration, providing both new and loyal customers a great shopping experience. Let Obermeyer's Drupal 8 integration with Drupal Commerce keep you warm because winter is coming ...

Happy 1st birthday Drupal 8!

I can't think of a better way to celebrate Drupal 8's one year anniversary than by sharing some incredible experiences that are being created with Drupal 8. Whether the project is big or small, features dinosaurs, or spreads awareness for an important cause, I'm proud to say that Drupal 8 is supporting an amazing array of projects. In my recent keynote at DrupalCon Dublin, I explained why the why of Drupal is so important. After one year of Drupal 8, it's clear how powerful our collective purpose, projects, and passions can be.

Thank you to everyone who has continued to contribute to Drupal 8! I can't wait for another year of exciting projects. Special thanks to Paul Johnson for crowdsourcing great examples that I wouldn't have known about otherwise.

Content and Commerce: a big opportunity for Drupal

Last week Acquia announced a partnership with Magento. I wanted to use this opportunity to explain why I am excited about this. I also want to take a step back and share what I see is a big opportunity for both Drupal, Acquia and commerce platforms.

State of the commerce market

First, it is important to understand what is one of the most important market trends in online commerce: consumers are demanding better experiences when they shop online. In particular, commerce teams are looking to leverage vastly greater levels of content throughout the customer's shopping journey - editorials, lookbooks, tutorials, product demonstration videos, mood videos, testimonials, etc.

At the same time, commerce platforms have not added many tools for rich content management. Instead they have been investing in capabilities needed to compete in the commerce market; order management systems (OMS), omnichannel shopping (point of sale, mobile, desktop, kiosk, etc), improved product information management (PIM) and other vital commerce capabilities. The limited investment in content management capabilities has left merchants looking for better tools to take control of the customer experience, something that Drupal addresses extremely well.

To overcome the limitations that today's commerce platforms have with building content-rich shopping experiences, organizations want to integrate their commerce platform with a content management system (CMS). Depending on the situation, the combined solution is architected for either system to be "the glass", i.e. the driver of the shopping experience.

Lush.com is a nice example of a content-rich shopping experience built with Drupal and Drupal Commerce.

Drupal's unique advantage for commerce

Drupal is unique in its ability to easily integrate into ambitious commerce architectures in precisely the manner the brand prefers. We are seeing this first hand at Acquia. We have helped many customers implement a "Content for Commerce" strategy where Acquia products and Drupal were integrated with an existing commerce platform. Those integrations spanned commerce platforms including IBM WebSphere Commerce, Demandware, Oracle/ATG, SAP/hybris, Magento and even custom transaction platforms. Check out Quicken (Magento), Puma (Demandware), Motorola (Broadleaf Commerce), Tesla (custom to order a car, and Shopify to order accessories) as great examples of Drupal working with commerce platforms.

We've seen a variety of approaches to "Content for Commerce" but one thing that is clear is that a best-of-breed approach is preferred. The more complex demands may end up with IBM WebSphere Commerce or SAP/hybris. Less demanding requirements may be solved with Commerce Tools, Elastic Path or Drupal Commerce, while Magento historically has fit in between.

Additionally, having to rip and replace an existing commerce platform is not something most organizations aspire to do. This is true for smaller organizations who can't afford to replace their commerce platform, but also for large organizations who can't afford the business risk to forklift a complex commerce backend. Remember that commerce platforms have complex integrations with ERP systems, point-of-sales systems, CRM systems, warehousing systems, payment systems, marketplaces, product information systems, etc. It's often easier to add a content management system than to replace everything they have in place.

This year's "State of Retailing Online" series asked retailers and brands to prioritize their initiatives for the year. Just 16% of respondents prioritized a commerce re-platform project while 41-59% prioritized investments to evolve the customer experience including content development, responsive design and personalization. In other words, organizations are 3 times more likely to invest in improving the shopping experience than in switching commerce platforms.

The market trends, customer use cases and survey data make me believe that (1) there are hundreds of thousands of existing commerce sites that would prefer to have a better shopping experience and (2) that many of those organizations prefer to keep their commerce backend untouched while swapping out the shopping experience.

Acquia's near-term commerce strategy

There is a really strong case to be made for a best-of-breed approach where you choose and integrate the best software from different vendors. Countless point solutions exist that are optimized for narrow use cases (e.g. mobile commerce, marketplaces and industry specific solutions) as well as solutions optimized for different technology stacks (e.g. Reaction Commerce is JavaScript-based, Magento is PHP-based, Drupal Commerce is Drupal-based).

A big part of Acquia's commerce strategy is to focus on integrating Drupal with multiple commerce platforms, and to offer personalization through Lift. The partnership with Magento is an important part of this strategy, and one that will drive adoption of both Drupal and Magento.

There are over 250,000 commerce sites built with Magento and many of these organizations will want a better shopping experience. Furthermore, given the consolidation seen in the commerce platform space, there are few, proven enterprise solutions left on the market. This has increased the market opportunity for Magento and Drupal. Drupal and Magento are a natural fit; we share the same technology stack (PHP, MySQL) and we are both open source (albeit using different licenses). Last but not least, the market is pushing us to partner; we've seen strong demand for Drupal-Magento integration.

We're keen to partner with other commerce platforms as well. In fact, Acquia has existing partnerships with SAP/hybris, Demandware, Elastic Path and Commerce Tools.

Conclusion

Global brands are seeing increased opportunity to sell direct to consumers and want to build content-rich shopping journeys, and merchants are looking for better tools to take control of the customer experience.

Most organizations prefer best of breed solutions. There are hundreds of thousands of existing commerce sites that would like to have more differentiation enabled by a stronger shopping experience, yet leave their commerce capabilities relatively untouched.

Drupal is a great fit. It's power and flexibility allow it to be molded to virtually any systems architecture, while vastly improving the content experience of both authors and customers along the shopping journey. I believe commerce is evolving to be the next massive use case for Drupal and I'm excited to partner with different commerce platforms.

Special thanks to Tom Erickson and Kelly O'Neill for their contributions to this blog post.

A plan for media management in Drupal 8

Today, when you install Drupal 8.2, the out-of-the-box media handling is very basic. For example, you can upload and insert images in posts using a WYSIWYG editor, but there is no way to reuse files across posts, there is no built-in media manager, no support for "remote media" such as YouTube videos or tweets, etc. While all of these media features can be added using contributed modules, it is not ideal.

This was validated by my "State of Drupal 2016 survey" which 2,900 people participated in; the top two requested features for the content creator persona are richer image and media integration and digital asset management (see slide 44 of my DrupalCon New Orleans presentation).

This led me to propose a "media initiative" for Drupal 8 at DrupalCon New Orleans. Since then a dedicated group of people worked on a plan for the Drupal 8 media initiative. I'm happy to share that we now have good alignment for that initiative. We want to provide extensible base functionality for media handling in core that supports the reuse of media assets, media browsing, and remote media, and that can be cleanly extended by contributed modules for various additional functionality and integrations. That is a mouthful so in this blog post, I'll discuss the problem we're trying to solve and how we hope to address that in Drupal 8.

Problem statement

While Drupal core provides basic media capabilities, contributed modules have to be used to meet the media management requirements of most websites. These contributed modules are powerful — look at Drupal's massive adoption in the media and entertainment market — but they are also not without some challenges.

First, it is hard for end-users to figure out what combination of modules to use. Even after the right modules are selected, the installation and configuration of various modules can be daunting. Fortunately, there are a number of Drupal distributions that select and configure various contributed modules to offer better out-of-the-box experience for media handling. Acquia maintains the Lightning distribution as a general purpose set of components including media best practices. Hubert Burda Media built the Thunder distribution and offers publishers strong media management capabilities. MD Systems created the NP8 distribution for news publishers which also bundles strong media features. While I'm a big believer in Drupal distributions, the vast majority of Drupal sites are not built with one of these distributions. Incorporating some of these media best practices in core would make them available to all end-users.

Second, the current situation is not ideal for module developers either. Competing solutions and architectures exist for how to store media data and how to display a library of the available media assets. The lack of standardization means that developers who build and maintain media-related modules must decide which of the competing approaches to integrate with, or spend time and effort integrating with all of them.

The current plan

In a way, Drupal's media management today is comparable to the state of multilingual in Drupal 7; it took 22 or more contributed modules to make Drupal 7 truly multilingual and some of those provided conflicting solutions. Multilingual in Drupal 7 was challenging for both end-users and developers. We fixed that in Drupal 8 by adding a base layer of services in Drupal 8 core, while contributed modules still cover the more complex scenarios. That is exactly what we hope to do with media in a future version of Drupal 8.

The plan for the Drupal 8 media initiative is to provide extensible base functionality for media handling in core that supports the reuse of media assets, media browsing, and remote media, and that can be cleanly extended by contributed modules for various additional functionality and integrations.

In order to do so, we're introducing a media entity type which supports plugins for various media types. We're currently aiming to support images and YouTube videos in core, while contributed modules will continue to provide more, like audio, Facebook, Twitter, etc. To facilitate media reuse, WYSIWYG image embedding will be rebuilt using media entities and a media library will be included to allow selecting from pre-existing media.

We consider this functionality to be the minimum viable product for media in Drupal 8 core. The objective is to provide a simple media solution to make Drupal 8 easy to use out of the box for basic use cases. This would help users of sites large and small.

Media library prototype
A work-in-progress prototype of the proposed media library.

Expected timeline and call for help

We believe this could be achieved in a relatively short time — to be included in Drupal 8.3 or Drupal 8.4 as experimental modules. To help make this happen, we are looking for organizations to help fund two dedicated code sprints. The existing contributors are doing an amazing job but dedicated in-person sprints would go a long way to make the plans actually happen. If you are willing to help fund this project, let me know! Looking to help with the implementation itself? The media team meets at 2pm UTC every Wednesday. I also recommend you follow @drupalmedia for updates.

I tried to make a list of all people and organizations to thank for their work on the media initiative but couldn't. The Drupal 8 initiative borrows heavily from years of hard work and learnings on media related modules from many people and organizations. In addition, there are many people actively working on various aspects of the Drupal 8 media initiative. Special thanks to everyone who has contributed now and in the past. Also thank you to Gábor Hojtsy, Alex Bronstein and Janez Urevc for their contributions to this blog post.

Updates from Dries straight to your mailbox