Use Case of AWS SQS
What is Amazon SQS?
Amazon Simple Queue Service (SQS) is a managed message queue service offered by Amazon Web Services (AWS). It provides an HTTP API over which applications can submit items into and read items out of a queue. The queue itself is fully managed by AWS, which makes SQS an easy solution for passing messages between different parts of software systems that run in the cloud.
Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. SQS eliminates the complexity and overhead associated with managing and operating message oriented middleware, and empowers developers to focus on differentiating work. Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available. Get started with SQS in minutes using the AWS console, Command Line Interface or SDK of your choice, and three simple commands.
What is SQS used for?
The most common ways to use SQS, and of course other messaging systems, in cloud applications are:
Decoupling microservices. In a microservice architecture, messages represent one of the easiest ways to set up communication between different parts of the system. If your microservices run in AWS, and especially if those are Serverless services, SQS is a great choice for that aspect of the communication.
Sending tasks between different parts of your system. You don’t have to be running a microservices-oriented application to take advantage of SQS. You can also use it in any kind of application that needs to communicate tasks to other systems.
Distributing workloads from a central node to worker nodes. You can frequently find messaging systems in the flows of distributed large workloads like map-reduce operations. For these kinds of operations, it’s essential to be able to maintain a queue of all the tasks that need to be processed, efficiently distribute the tasks between the machines or functions doing the work, and guarantee that every part of the work is only done once.
Scheduling batch jobs. SQS is a great option for scheduling batch jobs for two reasons. First, it maintains a durable queue of all the scheduled jobs, which means you don’t need to keep track of the job status — you can rely on SQS to pass the jobs through and to handle any retries, should an execution fail and your batch system returns the message to the queue. Second, it integrates with AWS Lambda; if you’re using AWS Lambda to process the batch jobs, SQS automatically launches your Lambda functions once the data is available for them to process.
How does SQS work?
SQS provides an API endpoint to submit messages and another endpoint to read messages from a queue. Each message can only be retrieved once, and you can have many clients submitting messages to and reading messages from a queue at the same time.
The messages that SQS handles can be unformatted strings, XML or JSON. Because SQS guarantees “exactly once” delivery, and because you can concurrently submit messages to and read messages from a given queue, SQS is a good option for integrating multiple independent systems.
You might well be asking: why use SQS if you can have an internal HTTP API for each service? While HTTP APIs are an accessible way to expose software systems to external users, it’s not the most efficient mechanism when it comes to integrating purely internal systems. A messaging queue is more lightweight. In particular, SQS also handles things like automated retries, preserving queue state across multiple availability zones in AWS, and keeping track of expiration timeouts on all messages.
The benefits of using SQS
For Serverless developers, using SQS generally provides a wealth of benefits, which you can read about below.
Scalability Your SQS queues scale to the volume of messages you’re writing and reading. You don’t need to scale the queues; all the scaling and performance-at-scale aspects are taken care of by AWS.
Pay for what you use When using SQS, you only get charged for the messages you read and write (see the details in the Pricing section). There aren’t any recurring or base fees.
Ease of setup Since SQS is a managed service, so you don’t need to set up any infrastructure to start using SQS. You can simply use the API to read and write messages, or use the SQS <-> Lambda integration.
Options for Standard and FIFO queues When creating an SQS queue, you can choose between a standard queue and a FIFO queue out of the box. Both of these queue types can be useful for different purposes.
Automatic deduplication for FIFO queues Deduplication is important when using queues, and for FIFO queues SQS will do the work to remove any duplicate messages for you. This makes FIFO queues on SQS suitable for tasks where it’s critical to have each task done exactly once.
A separate queue for unprocessed messages This feature of SQS is useful for debugging. All messages that couldn’t be processed are sent into a “dead-letter” queue where you can inspect them. This queue has all the usual integrations enabled, so you can subscribe to it using an AWS Lambda event, for example, to send a notification when an item can’t be processed.
CapitalOne Case Study
Capital One is a leading information-based technology company that is on a mission to help its customers succeed by bringing ingenuity, simplicity, and humanity to banking.
Challenge faced Before Integrating SQS
Before SQS Integration
After Adopting SQS
Benefits Of AWS SQS
redBus Case Study
redBus is an Indian travel agency that specializes in bus travel throughout India by selling bus tickets throughout the country. Tickets are purchased through the company’s Website or through the Web services of its agents and partners. The company also offers software, on a Software as a Service (SaaS) basis, which gives bus operators the option of handling their own ticketing and managing their own inventories. To date, the company says they have sold over 30 million bus tickets and has more than 1750 bus operators using the software to manage their operations.
The Challenge
The company previously ran its operations from a traditional data center by purchasing and renting its systems and infrastructure. In addition to the expense, several logistical problems evolved from this arrangement. The biggest problem was that the infrastructure could not effectively handle processing fluctuations, which had a negative impact on productivity. Additionally, the procurement of servers or upgrading the server configuration was an extremely time-consuming endeavor. Over time, redBus realized that a better solution was imperative — a solution that offered scalability to handle the company’s processing fluctuations. redBus looked to Amazon Web Services (AWS) for a solution.
The Benefits
Since migrating to AWS, redBus has seen measurable improvements in the bottom line. Padmaraju says, “By scaling up and down dynamically based on the load, we maintain performance as well as minimize cost. With the time savings that the IT and development staffs obtain from the AWS solution, AWS gives us an overall cost benefit of about 30–40%.” He adds, “By hosting at [the AWS Asia Pacific (Singapore) region], redBus.in gained significantly in terms of website performance by way of reduced latency (about 4x). This is a great advantage when the customers are from India.”
Of the many excellent characteristics of AWS, perhaps the most significant to redBus is the ability to “instantly replicate the whole setup on demand for testing by creating and destroying instances on demand for experimentation, thereby reducing the time to market.” Less time to market translates to increased profitability and success.
The travel agency anticipates expanding the AWS solution to include Amazon Simple Notification Service (Amazon SNS) and Amazon Simple Queue Service (Amazon SQS) for monitoring, alerts, and intercommunication. “Amazon SQS is an especially good solution for enabling messaging between external applications and our applications,” says Padmaraju.
Since joining forces with AWS, redBus has gained the freedom to experiment on new solutions and applications at minimal cost, increased the efficiency of its operations, and improved its profitability.
Coursera Uses SQS
Watch this video to find out how Coursera uses Amazon (AWS) to manage and integrate fast communication power over 300 free online courses from Stanford, Duke, Princeton, the London School of Economics, and other universities to more than 3 million students worldwide.
BMW Case Study
The BMW Group is using AWS for its new connected-car application that collects sensor data from BMW 7 Series cars to give drivers dynamically updated map information. BMW Group is one of the leading manufacturers of premium cars and mobility services in the world, with brands such as Rolls Royce, BMW, and Mini. BMW built its new car-as-a-sensor (CARASSO) service in only six months leveraging Amazon Simple Storage Service(Amazon S3),Amazon Simple Queue Service (Amazon SQS), Amazon DynamoDB, Amazon Relational Database Service(Amazon RDS), and AWS Elastic Beanstalk. By running on AWS, CARASSO can adapt to rapidly changing load requirements that can scale up and down by two orders of magnitude within 24 hours. By 2018 CARASSO is expected to process data collected by a fleet of 100,000 vehicles traveling more than eight billion kilometers.
Environmental Monitoring Solutions (EMS)
Environmental Monitoring Solutions (EMS) is based in Victoria, Australia. Launched 25 years ago, the company specializes in solutions that help petrol retailers gather and analyze data on the performance of their petrol stations. Its solutions provide remote monitoring and 24/7 support services — helping customers boost sales, reduce maintenance expense, and decrease the risk of accidents. Today, EMS operates with a team of 30 personnel.
The Challenge
EMS customers such as Viva Energy (Shell), PUMA Energy, BP, and 7-Eleven typically own and operate hundreds of petrol stations across Australia. The stations need to operate highly efficiently because profit margins are small. Yet, at the same time, they have to offer great customer experiences, ensure employee safety, and minimize their environmental impact. Sometimes accidents do occur, and a typical EMS customer is likely to incur annual costs of AU$15 million (US$12.13 million) for cleaning up underground petrol tank leaks or vehicle fuel tank contamination. To help customers maximize efficiencies while addressing the need for service excellence, safety, and environmental protection, EMS developed Fuelsuite, which enables customers to switch from legacy technologies that are largely manual and unify station management — significantly reducing costs. With Fuelsuite, customers can monitor inventories, deliveries, and prices. The solution also raises alarms in event of possible environmental incidents, such as underground tank overfills. After the successful launch of Fuelsuite, EMS focused on product development. It looked to connect sensors in the stations’ underground tanks and pumps and, regardless of the configurations of those sensors, collect all their data at 30-second intervals. The data would then be aggregated on a cloud-computing infrastructure and displayed via a web-enabled interface in Fuelsuite in near-real time. Russell Dupuy, founder and managing director of EMS, says, “Our job was to find an IT partner and cloud-computing provider that could help us re-engineer our Fuelsuite technology and deliver an innovative off-the-shelf product that was user-friendly and easily customizable.”
Why Amazon Web Services
EMS researched the market for cloud-computing providers, focusing specifically on providers of Internet of Things (IoT) technology. Through IoT, EMS would be able to collect all the sensor data and deliver it to the Fuelsuite interface. In 2015, when EMS first started looking at cloud providers and their IoT services, Dupuy found that Amazon Web Services (AWS) was way ahead of Microsoft Azure. “From what I’d read, I was more confident in the AWS IoT roadmap,” says Dupuy. “We were investing millions of dollars in this project, so I had to be sure of the provider we were going to work with.”
EMS then engaged with AWS Partner Network(APN) Advanced Consulting Partner DiUS, which organized a two-day workshop with key stakeholders to agree on a shared vision for Fuel suite. “The expertise that Di US has in AWS and IoT gave us a lot of confidence,” says Dupuy. Just as importantly, Di US agreed to the plan of designing, supporting productization, and launching the IoT-enabled Fuelsuite solution for a major EMS customer retail network within 12 months. “DiUS showed that it was possible to meet our schedule by rolling out the platform at first with just enough AWS IoT capabilities for customers, and then extending those capabilities down the line.”
DiUS and EMS designed the new version of Fuelsuite so that data from the sensors is aggregated by a custom-built physical device, called Fuelscan®, which is situated inside the petrol stations. Fuelscan then wirelessly transmits the data to the AWS Cloud, where it gets processed and delivered via the Fuelsuite web-based interface. So that EMS is able to offer customized functionality for different customers, DiUS designed the solution to take advantage of AWS IoT Device Management to securely onboard, organize, monitor, and remotely manage the Fuelscan devices. It also uses Amazon Simple Queue Service(Amazon SQS) to queue messages from the AWS Cloud to Fuelscan devices and relies on Amazon Elastic Compute Cloud (Amazon EC2) to process the messages to and from the devices. In addition, the sensor data is stored from the devices in an Amazon Relational Database Service Amazon Simple Storage Service(Amazon RDS) instance, and Amazon Elasticsearch Service is used to keep a historical record of all Fuelscan data. Finally, the collected data is archived in (Amazon S3) buckets.
The Benefits
At present, more than 1,000 petrol stations in Australia are benefiting from the IoT-enabled Fuelsuite solution. Station operators get near-real-time data on the performance of their stations, including how much petrol is being sold and how much is in the underground storage tanks. It also includes data on the pressure inside the hoses connecting the petrol pumps to the automobiles, and on the temperature and petrol level inside the underground tanks. “With our AWS IoT–enabled Fuelsuite solution, customers manage their petrol stations proactively rather than reactively — gaining a complete picture of petrol station performance to dramatically improve efficiencies and detect fuel leaks early to minimize environmental impacts,” says Dupuy.
The potential savings from proactive rather than reactive management are significant. Likewise, operators can use the performance data on fuel sales to optimize the distribution of the most heavily used petrol pumps. “We expect that customers gain a 500 percent return on their annual costs of the AWS IoT–enabled Fuelsuite solution,” says Dupuy. Furthermore, EMS expects the return on investment to increase over time. “The AWS IoT Core platform is extensible so that we can incorporate more data from the petrol stations.”
Dupuy says that without AWS and DiUS, EMS would have found it challenging to take advantage of IoT. “The expertise of DiUS and the total reliability of the AWS infrastructure meant we could develop and roll out our IoT-enabled Fuelsuite in 12 months and be sure it would meet the needs of our customers.”
EMS, in partnership with DiUS, is now developing Fuelsuite further, adding new services for the fuel industry. It will also be launching the product internationally. Dupuy says, “An advantage of AWS is that machine learning, deep learning, and other new artificial intelligence services can easily be applied to a customer’s aggregated data that is collected through the Fuelsuite solution.”
