Postscript, an SMS marketing platform, released its first AI product that allows marketers to test hundreds of AI-generated messages against a control message.
Postscript, an SMS marketing platform, provides marketers with the ability to A/B test their text messages, and new functionality is being released to help marketers streamline this process.
Postscript, an SMS marketing platform, allows marketers to send automated messages (called Automation Flows) to their customers. Recently, new functionality has been added that allows merchants to see previous versions of their flows.
There were several instances in the last year where the Flow Builder team leveraged the use of workshops in order to achieve a goal. Having led several sessions, I wanted to highlight a few important ones.
Workshops are used to solve important problems and generate ideas in a highly collaborate way. While similar outcomes can sometimes be achieved by an individual, including a group or team in the process can generate more and often better ideas. It is because of this reason that I led the Flow Builder team through two important workshops in 2023.
The first workshop I led was in response to internal feedback about a navigability issue in Flow Builder. There were some known spacing issues inside of the Flow Builder tool; however, the issues were amplified after a recent feature release. These spacing issues made it extremely difficult for individuals to navigate Flow Builder.
Unfortunately, our team didn't have the bandwidth for a complex solution; therefore, I proposed that we try a post-up exercise to generate possible solutions.
(Above) A video sent from an internal stakeholder
During the post-up exercise, each team member had a set amount of time to generate ideas on their own. Next, we sorted the ideas into broader categories and took additional time to vote on the post-its or categories that resonated the most with us. After voting, we discussed the ideas with the most votes.
(Above) The Figjam our team used to brainstorm ideas
Lastly, after the brainstorming session, I placed the most-voted on ideas in an Impact Effort Matrix. I then moved post-its around according to team feedback.
(Above) A matrix that organized the team's ideas based on effort and impact
Based on the workshop outcomes, the team was able to prioritize items that would create the most impact for the least amount of effort. One of these items included introducing a mini-map in Flow Builder. The workshop exercise helped reveal that the flow react library the team leverages for Flow Builder actually has a built-in option to add a mini-map. Therefore, very little effort was required to make the update.
(Above) The implemented mini-map
After introducing the mini-map, internal and external feedback concerning the navigability of Flow Builder was reduced by 99%.
Working for a remote company doesn't introduce many opportunities to work with others in person; however, the Flow Builder team was fortunate enough to meet up in person for a few days in August, 2023. During this time, we spent an hour and a half brainstorming ideas that individual team members had to improve the Flow Builder experience.
(Above) The Figjam prompt for the workshop
Similar to the navigability workshop, this workshop focused on first generating individual ideas and then coming back together for a group discussion. One unique facet of this session was the introduction of a “won't do” section. Instead of only generating ideas that the team thought would improve Flow Builder, team members also had the opportunity to push back on ideas that were already in our backlog or roadmap. While this section didn't get overly utilized during the workshop, the team had a few concerns to share about a couple backlog items.
(Above) General ideas and ideas related to AI enhancements organized into a matrix
After generating ideas and voting on them, the team came together to discuss the ideas that received the most votes. These items ranged from ideas that were strictly BE initiatives to items that were purely related to the content we surface in-app. Once we walked through all the ideas, I again created an Impact Effort Matrix and shared it with the team. (During our in-person collaboration time, we also ideated on ways to leverage AI/ML within Flow Builder. Those items were added to the matrix as well.)
Once the Flow Builder team helped to properly organize the matrix, the Product Manager, Engineering Manager, and myself reviewed and added some of the items to our stretch goals for Q4 in 2023. One proposed project (updating the message builder 3rd party library) even made it onto the Q4 roadmap as a priority rather than a stretch goal.
Merchants spend the majority of their time on Postscript, an SMS marketing platform, creating and managing their SMS campaigns and automations. Flow Builder allows them to accomplish this more easily by introducing multi-message functionality within a single SMS campaign or automation.
There is limited functionality and support for the current tools merchants are accustomed to using; however, merchants are wary of change and have been slow to adopt Flow Builder—in turn, missing out on improved features that can help make their jobs easier.
The legacy SMS campaign experience (below left) uses outdated design system components and has limited functionality such as a single-message restriction and lack of branching capabilities. Meanwhile, the Flow Builder experience (below right) supports single-message use cases as well as more complex flows that include important functionality like A/B testing.
As the lead Product Designer on the project, I interviewed merchants about their experience using Flow Builder. In addition to running customer interviews, I created a Pendo survey soliciting asynchronous feedback and utilized Amplitude to track Flow Builder usage. I also worked closely with Product and Engineering by conducting a workship to highlight opportunities for increasing adoption.
We made the following updates to the product in hopes of boosting Flow Builder usage:
(Above) The addition of a "Try Flow Builder" button added to the legacy experience
(Above) A step in the process for migrating legacy automations to Flow Builder
The updates we made to the experience resulted in an increase of ~700% for Flow Builder usage. We also saw ~25% of merchants who interacted with the “Try Flow Builder” button continue to create a flow.
In order to sunset the legacy experience, we are working towards product pairity between tthe legacy experience and Flow Builder.
Postscript, an SMS marketing platform, recently released new functionality (called Text-to-Buy) that allows customers to purchase products via the text thread.
Since Text-to-Buy is still in Beta status, it does not provide all the capabilities merchants are seeking. On average, our enterprise level merchants have 5 options per product that require their customers to make a selection (e.g. size, color, etc.), but the current implementation requires merchants to make this choice on behalf of their customers. Since merchants have a limited knowledge of their customers' personal details, they opt out of using Text-to-Buy in its current state.
Currently, the Text-to-Buy experience requires merchants to select a product and all its options before sending out a Text-to-Buy campaign.
(Above) What a merchant sees when creating a Text-to-Buy message
(Above) What a merchant's customers experience
As the lead Product Designer on the project, I iterated through several ideas and shared them with the Postscript design team for feedback. After sharing the refined designs with Product and Engineering, I used UsabilityHub to run a series of usability tests, comparing different approaches. I shared the findings with my team and led a kickoff meeting where I aligned with Engineering on requirements.
We ran two rounds of usability testing. For the first round, users ran through 2 mobile-simulated experiences on their computers. The first prototype had them place an order solely in the text thread, while the second prototype had users go back and forth between the text thread and a web checkout experience.
(Above left) The text-based prototype, (Above right) The web-based prototype
Since the results of the initial usability test were very similar, we decided to iterate on the two experiences further and test again. This time, we were able to test on mobile devices to ensure the most authentic experience for test participants.
The first prototype still took place entirely in the text thread, but the selection process was simplified furthered. Meanwhile, the second prototype was updated so that only the first step took place in the text thread and the rest of the experience was on mobile web.
(Above) The preferred text-based prototype
Once we had our kickoff meeting surrounding the project, we uncovered some possible limitations in the winning Usability Hub design. Therefore, as an MVP, we slightly shifted our approach by scaling back complexity and focusing on products that only have one customizable option. However, once we get validation from the new implementation, the team will spike on how we can leverage our data to make the winning design a reality.
(Above) What a merchant will see when creating a Text-to-Buy message
(Above) What a merchant's customers will experience
This project is currently in development, so we do not have any data surrounding our main objective around increasing Text-to-Buy usage. However, once the update is made, we will create product release materials to share with Postscript merchants and garner more feedback from the merchants that our currently using Text-to-Buy.
Homeowners that use Vrbo, a vacation rental platform, set property rules travelers must agree to prior to booking. These rules encompass items like max occupancy, minimum renter age, allowance of pets, and several other rules.
Hosts cancel on travelers due to confusion surrounding house rules. Vrbo has over 13,000 cancellations annually because of house rules. The result is unhappy customers and a loss of revenue.
There have been a lot of concerns surrounding the original house rules experience for both hosts and travelers. For example, one early study found travelers less inclined to book a property when custom notes were included in the house rules. (The outlined sections pictured above are areas for custom content.)
* A UX hueristic evaluation is an internal evaluation by subject matter experts applying the Nielsen Norman Group usability hueristics to an experience.
As the lead Product Designer on the project, I attended weekly progress meetings with the Product Manager, Technical Project Manager, Marketing Specialist, and Content Strategist. I also worked closely with the traveler designer to ensure there was consistency across the host and traveler experience. Lastly, I regularly shared my working designs at our host team design critique for transparency and feedback.
(Above) Some early iterations for a rule about pets
Our UX Researcher performed a remote moderated study where 9 participants walked through a multi-step prototype. Our team was looking for feedback on:
(Above) A brief look into the functioning prototype
After updating some of the content and syncing with the Engineering team about scope, we were able to push forward with an updated version of house rules.
The new host design utilizes components from the Vrbo design system, allowing hosts to always have the save action within view. The content in the host view is also clearly represented in the traveler view as the host intended.
The original rule included an "on/off" state as well as options for "yes" and "no". The new rule has only the "yes" and "no" options with other more specific inputs inspired by early user research.
In order to provide the Engineers with clarity, each rule has been broken down into its different states. Using progressive disclosure, we only surface relevant content when the appropriate rule choice is selected.
To improve the house rules experience even further, we plan to add more rules after implementing additional natural language processing for host custom rules and notes.
When using Vrbo, a vacation rental platform, hosts must sometimes change or cancel a guest's reservation if the need arises.
When a host cancels on a guest, everyone loses. The guest is disappointed, the host's search rank falls, and Vrbo loses out on revenue. Host cancellations contribute to 10% of reported guest pain points and current limitations of the reservations cancel and change capabilities lead to millions of dollars in operational costs—not including the revenue lost for Vrbo and hosts.
The original cancel experience has led to a lack of trust between travelers and hosts. In this experience, hosts are not required to explain to the guest why they are canceling.
The original change experience does not allow a host to add a payment request or issue a refund. These actions can be taken immediately following the flow, but there is little direction for hosts about how to handle these situations.
While we are currently working with the research team to test the newest cancellation and change flows, we have gained some insights from the swarm usability study:
(Above) A portion of the tested prototype that included changing a reservation
Though we are still waiting on the next round of testing to begin, the new designs are based off previous research and extensive feedback.
The new design adds friction against canceling at the top of the panel. We have also added a new required step of submitting additional context for every cancellation reason.
For host cancellations, we want to require 100% refund for guests since it is not their fault for these kinds of cancellations.
Though we may not be able to prevent all cancellations, we can at least help prevent future cancellations by prompting hosts to sync their calendars and avoid double bookings. This coaching is specific to the reason the host is canceling on the guest.
The options to change a reservation are now surfaced with the option to cancel. This allows for more scalability as change functionality grows.
The new design gives hosts the option to view their calendar availability when choosing to change a reservation's dates. In the past, host have provided feedback concerning how they prefer to view their availability via the calendar.
We have also included the ability to add a payment request or issue a refund within the change flow. The option presented will be based on a host's settings, but hosts are still able to override the selection.
While working on the vacation rental designs, we have also been working on designs for larger hosts like hotel owners. Our two products, vacation rentals and hotels, are headed towards convergence and the cancel/edit experiences lend themselves as an optimal place for consistency.
Vrbo is a vacation rental platform that allows homeowners to rent out their property. Homeowners/hosts can approve or decline a traveler's request to book their property. The decline flow allows hosts to explain why they cannot host a guest.
Host declines lead to disappointed travelers and a loss of revenue. Vrbo loses out on millions of dollars annually due to host declines. If hosts had an easier way to keep their settings and calendars up-to-date, a bulk of these declines could be prevented.
The initial decline experience provides friction for hosts—possibly too much with the red symbol and dramatic red text. This design attempts to scare the host out of declining, rather than persuading them to approve.
On the next step, hosts are bombarded with a confusing list of reasons. Once they select a reason, they are able to decline without any guidance on how to avoid missing out on future opportunities.
* A UX hueristic evaluation is an internal evaluation by subject matter experts applying the Nielsen Norman Group usability hueristics to an experience.
As the lead Product Designer on this project, I worked with the Product Manager and Content Strategist. We worked closely with the Engineering team to ensure our ideas were feasible from a code standpoint and received thoughtful UX feedback from the engineers in turn. I also shared the work-in-progress designs with the larger design team to ensure a variety of feedback.
(Above) A mapping of the old decline reasons to the newly proposed reasons
(Above) Different explorations prompting hosts to block their calendars if declining due to property unavailability
Based on user feedback, we were able to pare down the reasons for declining a reservation request. We also opted to add a step mid-flow to block a host's availability. Lastly, depending on the decline reason, we implemented coaching either mid-flow or post-flow to help hosts save requests or prevent future declines.
We used an A/B test to determine if the new designs had a positive impact on booking acceptance rates and a negative impact on booking decline rates.
After a few months, we were able to validate our designs and pushed the new experience live for all users.
Since the new experience has been live for all users, we have seen the following changes:
Since implementing new decline reasons, we have learned that many hosts are now selecting "Dates booked on another platform" (22%) when declining a booking request. This tells us that many hosts that list their properties on multiple websites are still not syncing their calendars between platforms.
With the new designs, we are exposing more hosts to the calendar sync functionality, but we still have a ways to go. The next step forward is to simplify the calendar sync process by exploring automation through app permissions.
While interacting with Vrbo, a vacation rental platform, hosts spend the majority of their time managing reservations and messaging guests via their inbox. The inbox page is the second most visited page on Vrbo.
The inbox is home to booking requests, guest messages, cancellations, and so much more. Therefore, it holds a lot of value. That being said, there is a lack of urgency with hosts responding to guests—especially when it comes to inquiries (messages without a booking request attached). 10% of Vrbo's revenue can be traced back to inquiries, but only 3% of inquiries turn into full-fledged reservations.
The original design treats every message with the same sense of urgency. There is no indication of unread messages or actions a host needs to take on a specific reservation.
* A UX hueristic evaluation is an internal evaluation by subject matter experts applying the Nielsen Norman Group usability hueristics to an experience.
During the aforementioned design swarm, our team was able to do a lot of competitive and analogus research pertaining to how other inbox-like designs were constructed. Utlizing the swarm artifacts, I worked as the lead Product Designer on the inbox redesign. I regularly met with the Product Manager, Engineers, and the Content Strategist to convey progress and troubleshoot issues with working designs.
(Above) Several early iterations of a mobile inbox design
Our UX Researcher performed a remote moderated study where 9 participants walked through a native prototype. We were looking for feedback on:
In the prototype pictured above, participants were able to send a payment request reminder from the inbox. Participants were also able to utilize native-specific functionality like swiping in order to perform certain tasks.
The new design not only highlights the booking status better than the original experience, but depending on the booking status, inline coaching maybe present to help hosts act more quickly when they need to. We have also added the estimated payout to inquiries and booking requests to help incentivize action.
We started designing the new mobile experience and worked up to desktop—leveraging native iOS components to make implementing more efficient for our Engineers.
Another way we hope to increase response and acceptance rates is by surfacing response and booking acceptance metrics to hosts. If a host's acceptance rate falls below a certain threshhold, we can coach them to do better.
This experience has not been implemented yet, but stakeholders are expecting a noteworthy impact in response times and acceptance rates.
AI Infinity Testing
Optimized A/B Splits
Automation Flow Versioning
Flow Builder Workshops
Flow Builder Adoption
Custom Variant Support
House rules
Decline coaching
Cancel and change
Inbox redesign