Interacting with Bulk Data in Force.com
Hi, my name is Richard Seroter. Welcome to this next module on Force.com integration APIs. This specific one is looking at bulk data interactions with Force.com. In the last module we took a long look at the REST API and saw how to use a web service interface to easilyinteract with Force.com objects and metadata, now we pivot a little bit to talking about bulk data and not doing things in a real time fashion. Specifically we’ll talk about some of the key concepts in the Bulk API to be aware of. We’ll review the lifecycle of a typical Bulk API interaction. We’ll see how in the world you authenticate against the Bulk API. We’ll review the job lifecycle, see how batches work, we’ll talk about transformation specs, which are powerful when you’re dealing with data that might not be sourced from Salesforce and you need to do some transformation. We’ll review how error codes work. We’ll look at executing bulk queries, which again, is a very powerful way to collect a lot of data without having to synchronously wait for it. And then finally talk about some best practices and limits before wrapping up. Like we’ve been doing, let’s define this. Really it’s programmatic access to quickly load data into, and retrieve data from, Salesforce. That’s why you use this, that’s what it matters. The whole point is how do I get data in and out of Salesforce efficiently knowing I’m dealing with a lot of data.
Key Concepts in the Bulk API
Let’s talk about some of the key concepts, three things I wanted to point out here. There’s the idea of jobs, batches, and records. So you process a set of records, which we’re familiar with in Salesforce so far, from the course so far, it’s really instances of s objects and that sort of thing, so you process a set of records by creating a job that contains one or more batches. The job specifies which object is being processed, what sort of action you’re performing, am I querying things? Updating things? Inserting things? And a batch then is a set of records sent to the server as part of HTTP POST requests. So you’ve got records contained in a batch that are defined by a job. All the batches within a job are independently processed, so it’s asynchronous for each batch, they’re all the sudden just processed, you know, it may happen somewhat serially or it’s going to happen in parallel, it’s really up to you, and each one succeeds or fails independently. And then finally you have multiple file formats accepted. I can send in CSV, I can send in XML, or I can even send in binary attachments like images or pictures that might need to be attached with individual set of records.
When Do You Use It?
When do I use the bulk API? Well three things I’ll point out for you. Data synchronization between systems, that’s your bread and butter here. That’s when I really want to connect data between systems in a relatively consistent but not real time fashion. I’m trying to synchronize my data warehouse with my CRM system, I’m trying to link my voter application system with a backend process, whatever it’s going to be, I’m doing data synchronization. Obviously this is ideal for one-time data loads. Hey I’ve just moved to Salesforce from a legacy CRM or whatever data driven platform, I need to do an initial data load over the weekend before I flip the switch and move everyone over. I’m going to use things like the Bulk API to make that happen. Now really when I’m transferring a significant number of records, if I’m creating 10 records Bulk API is a little unnecessary, right, I just showed you in the previous exercises in the REST API how it’s very easy to do bulk requests, so when I’m doing, you know, a few dozen records it might be better to do that via the REST API, but whenI’m transferring thousands of records, tens of thousands of records, it’s going to make more sense to do that in a thoughtful fashion using something like this bulk API. The key though is that this is data oriented processing, not process integration. So I’m not integrating processes between systems, I’m purely transferring data between those systems, whereas web services can kind of give the impression of invoking logic. In this case I’m truly just shunting data around.
Lifecycle of a Bulk API Interaction
Let’s talk about the lifecycle of a Bulk API interaction. Let’s walk through what this feels like. So I start off by creating a job, first thing I do via the API is create a job, define what object it is, define what I’m going to do with it, it’s a query, it’s an insert, what have you. Then I add batches. These might be CSV, they all have to be of the same type that you’ve defined in the job, but I’m adding a bunch of batches for the same record type and we’ll talk about how you define your batch size in a few minutes, but you’re going to add batches to that job. You’re going to eventually close it. It doesn’t have to be immediate, you have to do it at some point, and then when you close it you’re really saying it’s closed for business. I’m not going to do any more batches for this. Then I can check the status, and I can technically check the status while it’s still open as well, just to find out how processing is going. But I can check this status to see what’s going on, maybe I am finding out that I’ve processed one of the four batches and maybe I’m going to wait until this is done, until I retrieve the results if I queried it, and if I’m inserting stuff I’m using this to find out if I’m done yet. Then I can retrieve the results, the results on an insert may be 99 out of 100 succeeded, or maybe they all succeeded. If I’m doing a bulk query, this would be giving me back the actual raw data that I requested.And then I continue that loop again. I might create another job and do something else. So I’m following this cycle of creating jobs, adding batches, closing those jobs, checking the status, and retrieving the results, and at any stage I could actually call an abort as well, which will go ahead and prevent, you know, stop processing where I am right now, not allow any more processing for queued up batches.
Authenticating Bulk API Users
How do you authenticate Bulk API users? Well pretty straightforward, I acquire the session ID from the SOAP endpoint. All the same user, object, field security applies. So again, I’m using the SOAP endpoint though, I’m not using the REST or a custom one for bulk, same one for SOAP. And then what I’m setting is an X-SFDC-Session header, that’s where I’m putting that value I get back from the session ID, so it’s a custom header, I’m not using the built in authorization header in a SOAP payload or a session header. I’m not using an HTTP header that’s standard, I’m using something very specific to this.
The Lifecycle of a Job
So let’s walk through this flow a little more visually here. So some more detail. These are the flows, create a job, get job details, and others for each one. So creating a job, before I load any data I have to create a job. Job specifies type of object that I’m working with. This really defines, you know, if it’s a contact, if it’s a voter, what have you. What operation am I performing? Am I inserting? Am I querying? Am I deleting? So I’m able to choose what’s going on there. When I’m creating that I have some choice over is it parallel, is it serial and the like.Now this is called with an HTTP POST and we’ll do this in a few moments. I choose the object, I choose the operation I care about, I pick concurrency, it is parallel or serial? This is optional, I don’t have to provide anything there, by default it’s parallel processing. I choose a content type, is it CSV, XML, CSV with zip, XML with zip, with other objects, all those sorts of things. Those are things I get to choose. It’s represented by a JobInfo resource. You’ll see that come back in the web service request, this idea of a JobInfo, that’s actually the XML element that comes back. I use that JobInfo resource even when I’m creating a new job, getting the status for a job or even changing the status of a job, this sort of thing will come back over and over again. If we talk about getting the job details, this returns the job info and the state in XML. I called this with an HTTP GET. So get this, this is kind of a weird API to some extent where it’s a mix of SOAP and REST and all kinds of things, I guess it’s just making sure you’re paying attention. So in this case it’s kind of RESTful where I’m making a GET request to the job details versus using a SOAP credential and other things. Just don’t ask too many questions, go with the flow. But this shows things like processing stats, how many number of batches have I processed? How many have completed? How many have failed? How many are in progress? How many records did I process? How many retries have I experienced? Total processing time did it take to process all those records. So some good statistics come back from this as well. I can monitor a job within Salesforce. When I go into the setup screens in Salesforce I can actually see my job, I can see the batches that are running, I can see the usage quote, how much of my batch processing have I consumed of my allowable quote, and we’ll talk about that and the limits. I can list in-progress or completed jobs. I see a job history there, at least for so many days, I think 7 days. It also lists any in-progress ones. I have a live look at the jobs I’ve built.What’s cool is I can also review the request and response. I can actually see what was sent to the server and I can see what was responded back to the caller, so it’s a nice transparency here in the tooling. Any job or batch older than 7 days does get deleted and, you know, it doesn’t matter what status it’s in, but when I close a job I’m actively closing the job, I’m preventing it from adding new batches. I call this with an HTTP POST and the batch processing doesn’t wait for closure, it goes ahead and processes the batch even while the job is still open. As I mentioned, they’re deleted after 7 days regardless of status, so even I forget to close the job, it will inevitably get closed and deleted on my behalf, so you don’t have a full long audit trail here, you’d probably want to poll this data in to some other sort of data martif you need to retain the history of these jobs. If we look at aborting a job, that means no more records are processed. This is called within HTTP POST, but a key point is that there’s no rollback of previous changes. If I abort a job, no more records are processed, but any changes to data I’ve already committed are done and those aren’t rolled back, it just stops me from moving forward. If I actually wanted to rollback I would have to do another batch job that flipped the other records to another state or deleted them or whatever. So this is not all one big atomic transaction. When you abort you’re really just ceasing it from processing any further.
Let’s talk about batches, so a batch is a set of records sent via an HTTP POST. It’s part of a job. Each batch is processed independently by the server, not even necessarily in the order received, you’re not guaranteed it’s going to be in a particular order. This can be text or text + binary content. So I create this batch by submitting a CSV or XML representation of these records and any reference to binary attachments I happen to have. I can monitor it via the UI or the API. I can use the UI, as mentioned, and take a look at batches in progress,I can see what’s going on, or I can call, you know, the get status and batch info API calls to see what’s going on and see how well it’s processed, so lots of different ways I can track what’s happening here. There’s a lot of different states in the lifecycle. A batch can be queued, meaning the process hasn’t actually started yet. It can be in progress, meaning it’s processed, if the job is aborted, this batch will still complete. It can be in a completed state, that means that processing is done, you can retrieve any results, if some records failed, others could still succeed, it’s fine. It can be in a failed state, meaning there’s a server error. It can be not processed, meaning that the batch won’t process because the job itself was aborted before the processing began, so lots of different states that you would want to keep an eye out for the batches in your job. You want to consider the batch size, and we’ll talk a little more about optimization later, but ideally, you know, Salesforce gives you guidance that you start with around 5000 records, you adjust the batch size based on the processing time, because if it starts, if it takes more than 5 minutes to process a batch, you’re probably going to want to start splitting that batch up into multiple batches. You can start hitting time out errors as Salesforce won’t give you an infinite amount of time to process a batch. And then finally as we talked about, partial success is possible, right, so I can have failures. If individual records fail or the batch fails, what’s key is that when I submit that batch, any formatting errors and things aren’t reported when I send that batch in. I would see that stuff after the fact, so I just want to make sure that I resolve failures by looking at the records that succeeded, which ones didn’t, and if it did fail it’s going to tell me why. So it can still go back on you as the user to interpret the results, see if something failed, and handle that accordingly, however is appropriate, whether it’s to try it again or whether it’s to make some data corrections, or maybe to try to rollback those other changes by submitting a new batch that unwinds it.
Batches Take CSV, XML, Binary Content
Batches can take in CSV, XML, and binary content. So we will compare these two. CSV and XML files come in, I need to first look up the full field names. What is the name of the field? I’m not just going to put name if it’s actually called ID under the covers or voter name__c or whatever it is, I need to first of all know the full field names that I’m working with. The first row of a CSV actually contains the field names, so as for any comma separated file you would typically have a header row and so this would be the names all separated with a comma.For XML it uses SOjbect and field names, so you’re expected to provide the SObject and the field names within that. So you’re defining that very cleanly in XML format if that’s the format you’re passing in. The records all have to be the same object type; I can’t do a batch insert with a bunch of mixed objects. It’s all the same thing that fits in there. You have to include all the required fields or, as you can imagine, it will fail. So you have to know what things are required, make sure you do some up front scrubbing if necessary and data quality checks to make sure you’re inserting all the necessary fields so you don’t end up with a bunch of spotty failures or a lot of inconsistent results. If you’re updating missing fields are ignored, so unlike sometimes when you call certain sort of interfaces, when I leave a field empty or miss it, the server can interpret that as, well I guess I should empty it or delete it, that doesn’t happen here. So if I have a missing field, it’s not in that bulk request, it’s just ignored, I don’t assume that you want me to change the field that’s already there. Relationships can be supported, which is great. So if I have relations between objects I can model that successfully to some extent within the CSV and XML files. But there is no upfront validations, when I first submit this, if my data is garbage I don’t know that when I first submit it because this is an asynchronous interface, so you’ll find this out after the fact as these batches start processing and frankly failing. If I look at binary content, this is a zip file that has inside a request.txt file as well as a bunch of binary attachments. I might use this if I’m attaching a bunch of PDF contracts to a bunch of contract entities in Salesforce. Or let’s say people’s pictures, I’m attaching pictures of a particular precinct in our scenario. There’s a lot of reasons why we would want binary content in Salesforce. That request.txt file is a CSV or XML manifest, it’s describing what I want to do in that file. So that’s the description, we’ll take a look at one of those in a moment. And it does support nested structures, so I could have my images in a few different folders, for example, within that zip, and that manifest can define that sort of nested structure, if I’ve organized my binary content in a logical way.
Demo Description: Creating and Executing Jobs
So let’s do a demo. Let’s go ahead and review an XML batch of Voter records. We’ll create a new job for inserting the voters into the system. We’ll add a batch to the job that actually has the different voters in it. We’ll check the details of the job as it’s running. We’ll monitor that progress in Salesforce and see how it’s working. And then finally, we’ll close the job. So this lets us run through the full lifecycle of adding a bunch of voters into the system.
Demo: Creating and Executing Jobs
Alright let’s jump in and start creating some jobs, adding some batches, checking the details, closing the job, things like that. So what I want you to do, we’re in Postman right now, but let’s go ahead and jump into the code files for module 4. Within here, there’s a file called createxmljobbody. That’s what I’d like to do. Open that up, copy all the content to your clipboard, jump back into Postman, make sure you’ve got a new request XML, and I want to go ahead and paste that in. What I’m doing here is the jobInfo, as we talked about that’s the object, it’s an insert operation on Voter__c, and the content I’m going to pass in is XML. So we want the content type as XML because that’s the header we’ve already got on here, now I also need an xfdc session header, so let’s create another request and let’s copy in the SOAP login. Let’s make a SOAP request, because what I’m going to need is I need that value back from there, I need the session IDbecause that’s what the batch API uses here. So I’m going to copy that back into this request, and in here what I have is the X-SFDC-Session header, I’m just going to paste that value in. So I’ve got the body, now the only thing, of course, I’m missing is a URL. So what I want to do here is post that to whatever server comes back in your SOAP request /services/async/version, 35.0/job. So I’m posting to that particular address, and now I’ve got this body, I’ve got these headers, and I’m in good shape. So let’s go ahead and post this. It should create a brand new job. And I have a bad content type because I probably should have chosen application XML. Let’s try that again. Sure enough, I come down here and it tells me to create the job, it used the default parallel mode because I didn’t pass anything in, so it’ll do parallel processing, nothing new going on. Obviously nothing processing, but now I have a job open, that’s great. Let’s create another Postman request. This is one now where I want to add some batches to this. So let’s go ahead and take the URL from here, copy that, paste that over here, this will also be a POST here, so let’s switch to that. Now what I want to do is I have to process to job, but I need to put the job ID after this, so from this we got a job ID back, that’s this. Let’s copy that ID node value because that’s our job identifier. I’m going to put that after here and then I’m really posting to the batch, right, I’m adding a batch to this job, that’s what I’m specifying there.Let’s go ahead and add the right headers. Let’s switch this to application XML, and let’s also add the header for X-SFDC-Session, and I can go ahead and steal that from my create job query header, grab that token, stick that back over here. Great, so I’ve got all that. So now I’ve got everything set up, now I have to actually go ahead and jump down to body, and I have a few things I could do here, what I really want to do, I don’t want to paste it in here, I’m even going to go smarter than that, I’m going to binary. So I can actually pass this content in as the batch XML file. So let’s go ahead and choose the file. So I’m going to go to mod4 and what I want to pick here is the batch XML. Now let’s look at what’s in there real quick. I should have picked that so carelessly, so let’s jump into this thing, what I have is a couple of bulk voters, Voter 1, Voter 2. Now we know that precinct, there’s no way that’s good. So let’s jump back into Salesforce, and let’s pick a value from let’s say the 3rd District. I’m going to go copy that identifier, go find any sort of precinct you want, it doesn’t matter, and let’s go ahead and paste that in here so that we have a valid precinct for these new users. Do that, save that, now jump back into Postman. Let’s choose that file again, make sure it’s polling the latest copy, and then we should be able to submit the request. We have binary content; we’re sending this batch XML to the server. Now we see the state is queued. So the batch info is queued up to the server. So that’s great. So I’ve added a batch to the job and it’s probably already processing. Let’s prove that. Let’s do yet another request, and now I’m going to definitely want to save this whole URL. Let’s do a new request, let’s keep it as a GET, and now I’m going to do a GET against that job. I want to poll back information about that particular job. All we need to do here is add that X-SFDC header because there’s no body here. So here let’s grab the header value once again from another record, paste that in here, and then let’s just hit Send because it’s just a basic GET. so we have the job info, it’s an insert of Voter. We already have one batch completed, two records processed. Hey that’s pretty awesome. So it showed me that it was able to process that successfully. Let’s go ahead and jump into Salesforce and see if that’s true. So here back in Salesforce, we want to go look at Voters. If I jump into All, I don’t see these at the moment, so let’s go ahead and jump to Setup screens and check out our batch, a good excuse to go look at the monitoring. So if we go back to here I can do a simple search for bulk. Bulk Data Load Jobs, that’s good. Let’s go ahead and see what’s going on here. So what I have, I have two records failed, good to know. So I can see, here’s my job, I submitted it, it’s open, it’s an insert operation with Voter, 2 records were processed, but 2 records failed. So I don’t even know if I observed that in my payload that came back, so that’s interesting. Let’s find out what’s going on there. So if I click on the Job ID, I get some pretty cool details. I see the Job ID, I see when it started, I saw the processing time in milliseconds, and then I can jump down and see the batches. So I can see the request, I can see the response. If I want to view the result, it’ll show me, value of incorrect type, FIELD_INTEGRITY_EXCEPTION, alright so it didn’t like my precinct value. So I have to take another look at that and see what we did wrong, but it shows me very cleanly what happened there. So let’s jump in and let’s try to get a better precinct into our data, potentially it didn’t save my change from the last time. So let’s go ahead and go back to our request, and here’s where we added a batch and there’s a decent chance that for some reason it didn’t like what I did here, so let’s go ahead and pick some different types, now let’s pick batch ID again and hope that it likes this one a little more. Now remember, we changed that value, so let’s send that back in again. It’s queued. We can once again send the request, find out that more things are happening, two batches have completed. Let’s jump back into Salesforce. Let’s take another look at our Bulk Data Load Jobs. Ah-ha, so now we’ve got 4 records processed and 2 failed, which means these last 2 probably succeeded. Sure enough, here I’ve got 2 records processed and completed successfully over here. So let’s jump back, we should have two new types of voters that we add, there are two different voters. Let’s jump to All. Hey we’ve got the Bulk API Voter 1 and 2, those are horrible names, but at least it makes it really clear that it worked. So we’re able to see how we can go to those setup screens, see what happened, something failed, that’s no good, and we can actually see that happen. It told very clearly it didn’t understand that value I gave it for precinct, able to fix our data, make another request, and see it work. Now our job is still open, so let’s go close it. Let’s make one more request and this one is also to the same job URL, so let’s go ahead and save that. This is a POST request to that endpoint. Let’s go ahead and add the right headers, once again I need the X-SFDC-Session header that I will once again steal from here (Typing), and I need a proper job body here so I can access this. If we go to our file folder, you’ll see closexmljobbody. All you need to do is copy that, come back to Postman. We’ll do raw data, paste that in, this is of type application XML. And all I’m doing is changing the state of the job. So I’m simply saying, hey this job is good, I have to make sure I’m pointing to the right one, close this job, switch its state, same way I can abort it. Click Send. Now this particular job is showing as Closed and if I jump back into Salesforce I should see the same thing. If I want to go back to my setup screens I can click Setup Home, I can then go ahead and do another search for bulk. If I search for bulk I see Bulk Data Load Jobs, I should see this one, I can see its status is closed so I did the full lifecycle here and showed that I could add it, I could add batches, I broke one, which is always great for learning, I saw why, submitted a new batch into that particular job, and now I’ve closed that job. And so I’m finished, I couldn’t add anymore batches to it.
Using Transformation Specs
Your data isn’t always going to come in the format that you expect, especially when you’re doing bulk data imports. So how do you handle that? How you do import data in bulk that was exported from some other system? Do you have to fit it into the CSV format defined by Salesforce or the XML? Instead what you really want to be able to do is do map data field. So let’s say you have nonstandard CSV, meaning that it’s not the format mapped to explicit Salesforce.com field names, instead I want to make sure I’m defining something else. So let’s say you have something like this and you have a role of voters that comes in with Voter Name, Party, Precinct, and you want to map that. Well you can do that by creating a spec that gets scoped to a job. So these specs aren’t saved, they’re actually just tied to the individual job, they’re ephemeral, when the job is finished the spec is gone, but to upload that spec you can actually send a POST request to a specific URL that’s part of the job, and then as you update the batches, then you’re able to map that to that, and so that transformation spec has what you can see here, the Salesforce field name, like Name, mapped to the Csv Header in the data that’s coming in, in this case Voter Name, then you optionally have a Value field, so this would be a default value, for example look at the Political Party field where it’s mapped to Party and the default is independent if nothing is provided. And then finally, the last one is a hint, so this could help you, it’s also optional, helps you with date time interpretations, Booleans, things like that. That’s also something you could choose to include in the spec. You can see that the spec defines the mapping and when I apply that, then when I’m doing my data import it applies the spec and you’ll see the data come in in the correct format.
Demo Description: Creating and Applying Transformation Specs
Next up let’s do a demo. Let’s mess around with this a little bit. So we’re going to look at a non-standard batch file of Voters. We’re going to review the mapping spec that’s going to take care of this for me. We’re going to create a job, we’ll go ahead and submit that spec to the job, we’ll add a batch and the batch will reference this nonstandard format. We’ll go ahead and close the job, and then we should be able to review the import results and see if this indeed made it successfully.
Demo: Creating and Applying Transformation Specs
As we get started here with some nonstandard at “nonstandard content”, let’s take a look at our initial docs. So we have nonstandarddata.csv right here. If I open that, you’ll see I’ve got Voter Name, Party, and Precinct. And so Voter Name, Party, and Precinct refers, in this case, to Salesforce ID, we’ll obviously want to replace that right now just so that we’re safe and we’re mapping this to a known value. So let’s jump into Salesforce and put a known precinct value in here. Here in Salesforce I hopefully can jump to one of my recent records. This is the 4th District precinct. I’ll go ahead and copy that ID and I’ll plug that in. Now in real life you probably would not have the actual precinct ID in your source data, you’d probably want to figure out some other way to get that on the import or do something else like set a default value to this particular one to play it safe, but in this case we’ll go ahead and just be a little bit contrived.So I’ll save that, and that’s our source data. That’s our nonstandard format. Let’s look then at our spec. So spec.csv says Salesforce Field is Name, Salesforce Field is Political_Party__C and Precinct__c. The actual CSV value coming in there is Name, Party, and Precinct. And in this case, again, we’re defaulting it to Independent. So that’s our spec, that’s what we want to import as part of that. So let’s go ahead back to Postman, and in Postman if you hadn’t in the last exercise, I saved names for all of my jobs, we can reuse many of them here if we’d like to. So I can go ahead and let’s do another SOAP login. We’ll issue that call, get ourselves a fresh new session ID to work with here, save that, and then what I want to do is create a job. Let’s see, we want to find the create job or create batch, here we go. So in this case, we’re going to create a new job, it’s going to post to that job endpoint. We already have a header, let’s put the value in of our latest session token. Now we want to change the content type, so I want to change this to actually be text/csv because that’s what we’re going to send in now, we’re actually going to send in csv content instead. We’ll go ahead and change this to csv as well because, again, when we change this job we want to make sure that we’re sending in the right stuff. So content type is CSV, we’re dealing with the Voter object, and we’ll be doing an insert. Go ahead and click Send. We got a 201 created back, insert, job is ready, it’s open, everything’s running in parallel. So we’ve now opened our job, we’re ready to get started here. Let’s go ahead and add the mapping spec. So we know our job ID, let’s make sure we’re looking at that, we’ll need that value. Let’s create a brand new request here in Postman because now we’re going to want to POST to that location where the job is. So we’ll go ahead and go to where we’re going to add the batch to the job, this is from our old job, we’ll want a lot of this URL. And in this new tab, what I need to do is where we went ahead and created the job, let’s take the job ID, let’s go back to our new request, and we’re going to want to plug that in here and the interesting part is is that we’re going to hit that, its job, and then at the end of that we want to put spec, so we’re not putting a batch there yet, we’re putting spec. So we hit our server, Salesforce.com/services/async/version/job/job ID, and spec. Then we want to have the X-SFDC header or Session, and let’s go ahead and take the value we just used for this request because we know that’s a fresh token. We’ll copy that, put that into our new one. Excellent. So now we’ve defined the precursor here. Let’s go ahead and also add a Content-Type and this will also be text/csv, because that’s what we’re going to be sending in. Now let’s go ahead back to the body, and this is actually a POST, so let’s switch it to POST, so now I can edit the body. And it’s going to be binary. Let’s go ahead and upload that spec CSV file. So let’s choose that. We want to do spec.csv and let’s go ahead and submit the spec. Now we get 201 created. We don’t get anything back. Alright there’s nothing for it to tell me, but nowwe’ve applied a spec to our batch. So now let’s go ahead and add a job to the batch. Let’s come back to here and make sure we get the right job ID, so we can reuse previous requests, just be very careful that you reference the right job ID, so here’s a new job, and we’re hitting this /batch command. The header, we’ve got the content type as application/XML, we want to do CSV for this. Let’s make sure we have the latest session ID just to be safe, I believe I copied that over. Okay. And then what we want to do is I want to choose the file and we want nonstandard CSV. Okay so I’m passing it to job, here’s my current job ID batch, we already applied a spec, we already have the right headers, we’re sending in CSV, let’s make this actually text/csv. We’ve got our session and we’re sending the binary blob of the body here. Let’s click Send. Alright it’s queued. So I’ve gone ahead and submitted that, I’ve queued it. Hopefully it’s ready to go. Let’s go into Salesforce and see the status of this. If we go to the Salesforce setup screens, remember that we can check the status of our bulk job. If I do a search for bulk, I can see Bulk Data Load Jobs, I should have one that’s open right now. Sure enough I do. Three records processed. Let’s see if they were successful or not. Three records processed, none of them failed, that seems like a good sign. Let’s look at the result. I can view the result and I see some success messages, so that seems pretty good. Let’s go ahead and check and let’s go look at our records. So it looks like I probably had some success here, but nothing beats actually checking your data. Let’s see if we have the three new voters in here that we expect. Hey sure enough, we’ve got Chris Traeger, Mr. Gergich, and April Ludgate. So it did indeed add the three records, it did the mapping, and it took care of that. Now I’ve still got an open job. So let’s go ahead and close that. Let’s go ahead and make sure we take the right job ID, go to the one you’ve already closed before, we’re just going to pass a state of closed in there and let’s finally make sure our header is up-to-date, so I’ll grab the latest session ID for the closed job. Remember all we have to do is pass in the state=closed, it’s very straightforward. Let’s give it the right session, we’re passing in some XML, and the state is now closed. So now my job is closed, I was able to see that this successfully ran. I saw that request in sort of response. If we want to go back one more time to Salesforce, we can once more go back to those setup screens and what you can see is the result of the mapping. So first of all let’s confirm that this job is closed. Search for bulk, I see indeed my two jobs are now closed. If I check this one out, what’s also somewhat interesting is I can go view the request and what you’ll see is that you’ll see the mapped version. If we open this, you indeed see, I’ve got the Names, Political_Party, right, it mapped to it, and the Precinct. So the request I end up seeing is the one that got mapped because I did indeed have that mapping specification there.
Like anything, failures are possible. So how do I handle failures when I’m dealing in this sort of batch scenario? Well there’s a number of failures you can see when you’re working with the Bulk API. You can see ExceededQuota, this means you try to create more batches than you were allowed for the last 24 hours, and we’ll talk about that, what that limit is, in just a moment. You could get the InvalidBatch error and this means the batch ID you provided is invalid. You could get InvalidJob, this means you gave the wrong job ID and that doesn’t work.You could have an InvalidUser, maybe you don’t have the right permissions or the job or batch specified was created by another user and you don’t have rights to it. You could definitely get Timeouts and, again, if you get this, this is when you should split those batches into smaller batches, there’s a 5 minute limit for processing 100 records. In addition, if it takes more than 10 minutes to process the batch, the Bulk API actually puts the rest of the batch back in the queue for later processing and if it keeps exceeding that 10 minute limit on these next attempts, the badge is put back into queue up to 10 times and then it’s just marked as failed. So you can end up stuck if it hits these timeouts too often. Then finally you can also have, or in addition, you can have too many locks, and this will happen if you have too much contention with your updates, we’ll talk about that in a moment as well. And then no good set of failures is complete without the unknown.This really means, as you can imagine, who knows what happened, and so hopefully you can try again and get something better.
Executing Bulk Queries
Let’s talk about bulk queries. This could be a thing you don’t think about with a Bulk API, but this is actually a pretty cool scenario for using it. Really the idea of reducing the number of raw API requests by being more efficient, by grabbing large data sets with a single request. You can pull down up to 15 GB of data by a query, now it gets divided into fifteen 1 GB files, but you can poll this back in both XML and CSV format. You’re getting these SOQL requests in the HTTP request itself, so you’re actually sending the raw query as part of the request, then you’re getting later back the results. What you get back is this result ID. So when that bulk query is processed, Salesforce runs the query, it goes ahead and gives you things back and you get back a result ID, which then you can use to go retrieve the rest of the data. So I take that result ID and then I say, okay give me the response, you’re ready for me. So when it’s done it gives me that result ID. You have a pretty good subset of SOQL stuff supported here, but there’s things that aren’t supported. You can’t do things like count, rollup, sum, groupBy, offset, nested queries, and relationship fields. So there’s some good stuff in here, but don’t expect that this is the full extent of the Salesforce query language.
Demo Description: Submitting Bulk Queries
Our final demo here is looking at creating the job for a query operation. Submitting the query batch to retrieve a bunch of Voters. I want all the Voters back in a single query, maybe that would be too big, if this was thousands or tens of thousands, to poll back in an API via REST or SOAP, I want to use the Bulk API instead. We can monitor that batch status, we can retrieve the result key, then pull down the actual CSV and check out the results.
Demo: Submitting Bulk Queries
Alright so here’s our last demo of this particular module. What we want to do is actually submit a bulk query and pull back a bunch of records. So first off, let’s go to the folder that has our various data in there for this module. And something you’ll see here is bulkcreatequeryjob.txt, so let’s open this thing up, copy all that, and what’s in here is its query for Voter__c of type CSV. We already have something similar to this. Let’s jump back to Postman and if we look at the create batch, let’s go up to the top here, if we look at the body, we already have something pretty similar to this. We’ll just go ahead and paste in the new and what you see is the operation is query, very important there. My header should still be good because I’ve done this recently. If you’ve taken a little time between sessions you might want to go retrieve a new session from the SOAP API to get back your token. So let’s go ahead and create a brand new job. I got back a 201 created as well as a payload with a new job ID. So what I want to do now is create a bulk query. Alright so here’s my ID for my particular job. I’m going to go ahead and add a batch. Alright pasted that in, my session ID should still be okay, and this is also going to be a text/csv, which is great. I’ve got my session header there already, but now most importantly I want to go down to the body, we’re going to switch back to raw, we’re going to delete all of this, and now we’re going to submit to this job, but we want actually to submit some raw content. So what I want here is just type in SELECT Name, Precinct__c FROM Voter__c. So we’re just sending really a raw request over, raw SOQL query over in the message. Let’s go ahead and submit that. Once again we got a 201 created back and it’s queued. Great. So let’s go ahead and jump over to Salesforce and see if this is running. Here in Salesforce let’s jump to the setup screens. Once again, do a search for bulk, there’s our Bulk Load Jobs, we should have one open right now. Here it is. We can see 11 records were processed, that’s exciting stuff. Let’s look into the details. You see I can even abort it from here if I want to. Sure enough it processed 11 records, so now we know this technically ran, so what we want to do is create a request to query the results back. If I go to Postman, what I want to do is let’s use a new tab and let’s steal on of the IDs that also has the job in there already. So we’ll get this one where I added the job to the batch, or batch to the job rather. Let’s go ahead and this is going to be a GET. Let’s make sure that I’ve got the right values here, this is the one I just queued, here’s the ID of this particular batch, we already have the job ID in here, and so now what the URL is it’s job/job ID/batch/batch ID, and then result. That’s the value I want. So let’s go ahead and add the X-SFDC-Session header and let’s make sure we get a good value for that in our header. Let’s paste that in. And now we’re submitting a request for that particular batch, I want the results for that particular batch that’s in that particular job. We’ll go ahead and send, and I get back a unique ID. So this is now the result ID. I can use this to retrieve the results. So in the same request, let’s go ahead and just stay where we are, so in this I can take this same one. I have the result and now I can actually do whack result ID. Again, very RESTful like URL in some extent. So if I now send this, what I get back is the raw content. I’m getting CSV content back. So here I am getting, if I look at the raw, I can see I’m just getting Name value, it’s comma separated, and I’m getting all of my data back. So here I did, I got the full data back by hitting my job, then my job details, my batch, my batch ID, results, and the result ID and I can get back the data. So you can imagine this is a great way to get back lots of information by submitting an asynchronous request, checking the status of it, retrieving the result ID, and then actually retrieving the results.
Best Practices for Bulk API Usage
Let’s talk about a few best practices for using this Bulk API. There’s definitely some tips here. So using parallel mode wherever possible. So you’re going to get the most benefit from this API when you’re running these batches in parallel, that’s the default mode, which is handy, and it’s going to obviously help you load this data faster. You’re letting Salesforce decide how to run all these concurrent jobs and load allyour data. But you have to be careful because on one hand this parallel processing could actually cause lock contention, and we’ll talk about that in a moment. So if you’re concerned about that, you could process using serial mode, and this is going to, obviously, be slower, but it may help you avoid time outs and things like that. So you want to proactively try to avoid this lock contention. If you load a lot of batches and they all contain references to the same account or feel or record or what have you, they could all try to lock the same record itself. And so if you do that, you’re going to experience a lock time out. And Salesforce is smart, it’s going to try to do retries and hold that lock and then it’s going to retry to see if that lock is free, but at some point it’s going to have to give up. And so you want to make sure you try to avoid heavy operations against the shared object. So it also is good to know which operations cause lock contention. So whether you’re creating new users, whether you’re updating ownership or updating user roles, updating hierarchies, there’s things that can cause lock contention, so understand those. You’re going to want to minimize where possible the field count. As you can imagine, don’t upload data you don’t need to. It’s always going to be faster if you have fewer fields loaded for each record. Every time you add things like lookup relationships and foreign keys and rollup summaries, you’re going to clearly increase processing time. So you may not be able to control that, you may just be dealing with that, but where possible be aggressive about sending in the least amount of data possible on an import. And then finally, where possible minimize your post-processing activities, things that when you upload data if it triggers workflow actions or triggers or things like that, it’s clearly going to slow you down and, again, you may bump into that time out limit if you’re doing some really intensive things every time you upload data.
Bulk API Limits
Let’s talk about some of the limits. There’s a few things here. You have 10 minutes to process a batch. Important to know, I mentioned earlier you have a 5 minute limit to process 100 records, and if it’s more than 10 minutes you’re going to see the remainder put back in the queue for more processing. You can do 5000 batches per rolling 24 hour period, so you can create new batches associated with a jobthat’s over 24 yours period, but you can only submit up to 5000 batches per rolling window of 24 hours. As you’re thinking about maximum record count, you can have a maximum of 10,000 records in a given batch, that’s also 100,000 characters. You can have a maximum of 5000 fields, and then you also have the maximum size for a single CSV or XML file at 10 MB, so that zip file, that content can’t exceed 10 MB. If it’s unzipped it can go up to 20, but in essence this is, again, where you think about your batch size and you think about your batch content and try to arrange things accordingly. Now there’s also some limits on retrieving, so the maximum retrieve file size is 1 GB and there could be a maximum of 15 files. So you could get a total of 15 GB of data back. So just be aware of these limits. Sometimes you have to work within them and it’s good to know them ahead of time versus trying to kick off some big things and getting frustrated when you can’t get your responses back.
Bulk processing isn’t as hot and interesting as so many new real time processing and stream engines, but so many things matter still with bulk in that you have to do bulk data processing, you have systems that demand that, you have legacy apps you’re integrating with you don’t need the vigor of a real time integration, so bulk still makes a lot of sense. We did an overview here. We talked about those key concepts so you understood why Bulk API matters. We went through a little bit of the lifecycle of creating jobs and closing them, adding batches and the like. We authenticated Bulk API users, thanks to SOAP. We talked about the lifecycle of an individual job, and then the batches within the job, and understanding that relationship of a job has batches, which contain records. Sometimes you don’t get the initial spec that you expect, so you need to be able to transform that into a format that is able to be handled by Salesforce. We talked about some of the error processing. We did some execution of bulk queries, which is some pretty cool functionality. We talked about best practices and limits. So hopefully this gave you a good sense of the Bulk API and how jobs and batches and records all work together to help you do both insert, update, delete sort of actions, as well as retrieve actions in a more efficient fashion. Now we’re going to pivot completely back into more of a real time integration scenario next up with outbound messaging.
Using Force.com Outbound Messaging for Real-time Push Integration
Hi there, my name is Richard Seroter. We are in another module here in this course about Force.com integration patterns and APIs. In this module we’re going to talk about outbound messaging and real-time push notification. In the last module we looked at batch processing, how we take a deep dive into real-time integration, so a little bit of back and forth, but hopefully this gives you a good sense for the sort ofbreadth that you have available in Force.com. In this module we’re going to talk a little bit about reliable delivery, set the table, talk about some of the key concepts you need to be aware of for outbound messaging. We’ll review how you secure outbound messaging based solutions. We’ll talk about the flow for creating an outbound message, the interesting parts of creating a client to receive an outbound message, how in the world you track these messages and follow up on what’s going on, and then finally a quick summary. For definition purposes, outbound messaging is how you react to data changes by triggering messages to internet-facing endpoints. All those key points we’ll address throughout this module, but the key is it’s event driven, it triggers a message, and I goes to an endpoint that you operate somewhere that is internet accessible, that the Salesforce can reach it.
About Reliable Delivery
Let’s talk about reliable delivery, it’s an important thing because as you start debating which APIs to use, reliable delivery is going to play a part in your thinking. The key is is that most communication today is happening over unreliable channels, specifically HTTP. And so much of our interactions between systems now depend on what is inherently an unreliable channel, so you end up building up some machinery and process around that to try to confirm delivery and make sure that you get some at-least-once delivery, and that’s something to think about, only once or at-least-once is difficult, but at least possible. Exactly once is very difficult, you’re rarely going to have systems that can do exactly once delivery, instead you have at least once, only once, things like that. So you have a few things you can work with, but you should expect at least once delivery where messages could be delivered multiple times, that’s what you want to try to do. In some cases you may get, you know, at most once in some systems that haven’t built up that machinery to try to make themselves reliable. How you do this is typically through asynchronous queuing, so how can I have something call on the HTTP endpoint if that endpoint is down or if something happens in transit, how do I confirm that I’m able to retry it? And typically that is done through queuing where the request is queued up, if it’s not able to succeed, and retried later. It’s not all done in memory at the same time, instead you’re relying on something that can store and forward the data later on.
Key Concepts in Outbound Messaging
Let’s talk about some of the key concepts to think about when working with outbound messaging. So these are triggered by workflow rules based on field changes. So these workflow rules, watch for specific kinds of field change and then they trigger an automaticSalesforce action. Now these could be things like sending an email, creating a task record, or sending an outbound message. These messages are then sent to an endpoint that you specify and contains whatever field you’ve chosen when you create the outbound message. Then when that endpoint, whatever it is, takes that message in, it does whatever it’s supposed to do with it, it’s asynchronous and processes as you’d expect and gives you back an acknowledgement. The message is reliably sent to that endpoint because it’s queued locally, it’s not just the process sees it, sends it out, hopefully your system is online, if not the message is dropped. Instead it’s queued locally and a separate background process within Salesforce actually performs the sending, and so it gives you a reliable messaging. Now it’s not infinite, there’s still a time out and only so many retries that the system will try, but you get a much, much higher level of reliability than a sort of ephemeral it sends it out there and hopes it makes it. In order to listen on this, you actually have to build a web service endpoint that’s specifically SOAP based, that can implement the web service definition that Salesforce expects and so your listener implements that WSDL so that the message that comes across from Salesforce can get properly interpreted. What’s pretty interesting, and I really like this scenario personally, is the support for callbacks because in essence you have three options for integration solutions powered by outbound messaging. You can accept the risk and integrate solely with data pushed by outbound messaging, saying that’s how I link my systems is if it comes in an outbound message I update my system, if it doesn’t I don’t. Alright, number two is I have periodic Bulk API sync process that compliments outbound messaging and ensures that systems are in sync, so let’s say I’m always hooked up through outbound messaging, but every night at midnight I do a Bulk API call and synchronize data and maybe catch anything I may have missed during the day, kind of a fallback option. A third option is saying, I only send events via outbound messaging and I pollthe data live via a callback because when I send an outbound message I can choose to include a valid session ID so that the receiver may have no data in there, all they do is have a message that says an object got created, a voter got created, now go call back and poll that data live. So in that case even if there’s 10 requests I’m always going out and polling the latest, I’m not dealing with stale data, so callbacks are a cool way to be event driven, but not be beholden to the data in the event, you’re only getting a notification instead of actually getting a payload that you have to process. So I like that pattern a lot.
When Do You Use It?
So when do you use outbound messaging? Well a few scenarios, you want to replace polling-based solutions. Nothing is more frustrating to a service provider than clients how call their API every 5 seconds asking is anything new? Is anything new? Is anything new? I really don’t want my systems to behave like a 3-year-old. Instead it’s much nicer to have a solution that pushes things, yes here’s something new, process it. It’s so much less wasteful, it’s so much more practical, but you have to have a lot of faith in that solution that can push it reliably, but hopefully you can replace many of your polling-based solutions. And that’s the key is you’re creating a real-time response of cloud system. Having this sort of organism where data is being synchronized in real-time across these environments versus constantly trying to poll and synchronize, data is being pushed out as things are changing. And you may use this to trigger business activities. You might want to say, look when this thing happens, this particular deal changes state, you know, it moves from ready to closed. You know what, kick off an activity that sends some things between systems, it treats you more into an event driven model where I can trigger activities based on other things happening.
Securing Outbound Messaging Solutions
Let’s talk about securing outbound messaging solutions. What does that mean? So you have a few things in your disposal here. Typically when you send this you would do it over SSL, so actually you can configure the require of that and so you can make sure that your sending an SSL based transmission, so at least in transit your data is secure. You can also whitelist the Salesforce IP address in your client listener, so you could know that someone’s not trying to spoof this and hit your outbound messaging endpoint, you could make sure that you’re doing IP address filtering and only taking it if it comes from Salesforce. What’s also nice is you can validate it against a Salesforce certificate. So Salesforce shares with you the Salesforce client cert, and then you’re able to use that on your application side to make sure that it matches what came across in the outbound message and wasn’t tampered with along the way.
Flow for Creating Outbound Messages
Let’s talk about the flow for creating an outbound message, what happens is you click an option to create an outbound message, you choose which SObject this applies to, you set the name and the endpoint URL that you’re going to listen in on, you select which fields you’d like to send across in the message, you save the outbound message, and then you associate it with a workflow rule, so something that’s going to trigger this message to occur. You have another option too where you could choose to create the workflow rule first and then create an outbound to tie to it or you could create an outbound message first and then attach to one or many workflow rules. So that relationship is somewhat loose between a workflow rule and an outbound message, it can be kind of many to many.
Demo Description: Creating an Outbound Message
So let’s go ahead and do a demo. What we’re going to do here is first off create an outbound message for the Voter object. What I really want to do in this scenario is every time we contact a voter, I’d like to send a push notification that updates a map in our application, so I can see live real-time when we’re talking to customers. Imagine that being up in a big screen in the precinct office and I could see things updating live whenever I’m talking to a voter or one of my field people has knocked on the door and contacted somebody. So we want real-time message synchronization and visualization. We’re going to set the placeholder URL, we’re going to choose which fields we’d like to have come across here, we’re going to associate it with a workflow rule. So the first thing we’re going to do is create this outbound message, later on we’ll build the client. Let’s get the first part done here.
Demo: Creating an Outbound Message
Outbound messages are a lot of fun, let’s go ahead and create some. So jump into the setup screens here by clicking the little gear, going to Setup, and we can find this in a couple of ways, in the new Lightning Experience I can go to Process Automation, scroll down so you can see that a little better. Once I do that you can see Process Builder and then you see Workflow Actions, and you see Outbound Messages here. So let’s go ahead and click that. You see I could also start it with the workflow rule, instead I’m going to go ahead and start with Outbound Messages, and what I’m asked to is create a new one. Now we’ll get to here in a moment where I can view the delivery status and things like that. Let’s start by creating a new outbound message. Which object do I want to work with? It’s a single object, so I can’t span objects with this, let’s go ahead and pick Voter. Click Next. Now I’ve got a few choices here. So let’s go ahead and call this Voter Contacted OM, outbound message. Endpoint URL, now I don’t have this yet, right, we haven’t deployed a listener yet. So I can temporarily just put localhost here, and you can choose which user you’d like to send this as, so I’ll send this as me. Now as I mentioned, for callback scenarios I could choose to include the session ID, that would mean maybe I want to send no data here and I just want the customer or the client to call back whenever they get it and poll the latest and greatest. The only thing you should send is the ID, that could be fine. But we’re not going to do that here, we’ll do contact 2016, I want to know that, that’s going to be a value I care about. We have the ID coming. I want the Last_Contact_Date that we created as a custom field. I want the Mailing_Address__c, and I want the person’s Name. So those are the fields I want to come across. I don’t need to send everything, those are the only things I care about here. With that said, let’s go ahead and save this. Now I want to associate this with the workflow rule. So once I have this, I’ll go ahead and see all the outbound messages that I’ve created. Here I see the details. Here I can get the WSDL, which we’ll use in a few moments. I can see which fields to send. I could edit this to send the session ID or edit it for other reasons, edit the endpoint URL. Right now this isn’t very good because it’s not going to send it anywhere, but it will once we update that value. I can also see it’s not used by any workflow rules, it’s not part of any process, so let’s change that. Let’s go to Workflow Rules over here on the left, once again I get some information, I think I’m done seeing that so that should be good. I’m going to create a brand new workflow rule. Let’s go ahead and associate with the Voter object again, Next. The Rule Name may be Voter Contacted Rule and I want to do this when created and edited. And I want to pick the criteria, so I can choose the criteria, in my case it looks at the values. I can say whenever the Voter Contacted (2016) equals and I can pick from here and it’s a Boolean, so I should be able to choose between true and false, insert it. So whenever Voter Contacted (2016) equals True, I want this rule to fire. So let’s click Save and Next, and then I have a choice to add the workflow action here. So I could add a new outbound message or I could pick an existing one. So I’ll pick Select Existing. Here’s the Voter Contacted, we’ll click Save, and now I see that I’ve got theoutbound message, it’s tied to this, so when this rule fires it’ll be ready. I can also see, I want to make sure my rules are actually ready to go. Voter Contacted, so I could activate that whenever I’m ready, I can see it’s tied to that criteria, I can see here is the rule, and whenever I’m ready I could go ahead and activate, which we’re not yet. We will in a few moments. So in this exercise we created an outbound message, we chose which fields we wanted and then we created a workflow rule and tied that to it.
So next up we need to actually configure the client to receive an outbound message. What you get is the web service definition for that outbound message. Can you imagine what’s in there? Well what’s in there is going to be the object you chose, in this case Voter, and it’s going to be whichever field you selected, that’s going to be the WSDL, that’s what you have to implement, so instead of consuming it and calling that endpoint, instead you’re implementing it so something outside can call that in, so you’re implementing the service definition.Now if you’re using things like Java and .NET and others, they actually have tools to be able to pass in a WSDL and generate the stuff interface that you need to implement. What’s very important is you have to configure an acknowledgment as well, you’re not just receiving the message, you’re also sending back a very simple true or false acknowledgement that says I got this or not. It didn’t say I process it well, it didn’t mean that it was a successful processing downstream, it simply means yep I got it, and so what Salesforce uses that for is to say, do I try this again or was this successfully transmitted, and then it takes it out of the queue and it’s good. If it doesn’t get that acknowledgement back it will send the message again because it will assume it failed to make it. So it’s very important to configure that and send it back, otherwise, you’re going to keep getting pummeled with additional messages. Finally you need to make sure you deploy this client to an internet-accessible host. Salesforce has to be able to reach it. It can’t be sitting behind your firewall, it can’t be walled off. It needs to be internet-accessible, which is why you use those different security strategies to make sure you’re not inviting traffic or nefarious activity.
Demo Demonstration: Configuring a Client
What we’re going to do now is actually build a client or I’ve built a client for you, we’ll review that endpoint WSDL, we’ll look at the Node.js application and how it implements that particular WSDL. We’ll deploy the application to a cloud host, I don’t care which one, I’m going to be using a particular one, you could push this anywhere, it could be any sort of cloud server, just pick one. So I’ll deploy it to one, you could deploy it anywhere, and then we’ll trigger the outbound message and actually see it come into the system. Let’s jump in and actually see one of the more exciting things actually take effect in a Salesforce environment.
Demo: Configuring a Client
Tracking Outbound Messages
I hope you enjoyed that demo, it’s one of my favorite ones that I built. I have low standards I guess, but this is something I really enjoy. It’s fun to see real-time things happen. So that’s one that sometimes it’s a very visual way to understand why real-time sync is awesome. So how do I track outbound messages? How do I make sure that things got where they were supposed to? So I have a few different options here. First I can view the items queued for delivery. I can see a page that shows me what things are queued up. Now obviously you saw things happen so fast that rarely can you actually catch it in the queue because Salesforce is so great at sending things out. But what’s nice is I can observe failures, so if it didn’t make it I can go ahead and see things get queued up there, and that’s really the best example of seeing how this sort of reliable messaging works. From there I can edit or delete items in the queue. So it’s not just a read-only queue that I’m at the mercy of watching and hoping that it refreshes, I can actually manually retry or wait for the automatic interval, but I’m not powerless, I’m able to see what’s queued up, I’m able to make changes if necessary, and I can even shove it back through if I know my downstream system is back online.
Demo Description: Tracking Failed Outbound Messages
In this last demo we’re going to go ahead and test that out, so what I want to do is disable my client app, stop it, right, simulate real life where systems go offline, there’s connection blips, there’s things that can go wrong, which challenge your assumptions around reliable delivery. So let’s go ahead and disable the client app, we’ll trigger a new outbound message, then we want to go ahead and observe that being queued up or re-enable the client application, again, hopefully just like real life, and then manually resume that outbound message and see how that flows through and see how I don’t lose data just because there’s challenges in my system communication.
Demo: Tracking Failed Outbound Messages
I am back in my development environment and I’m looking at the map that we had just used earlier to send these outbound messages. Now we want to simulate failure. We want to see how Salesforce responds to that. So let’s go ahead in AppFog, now my application is still online, but I’m going to go ahead and stop my application. So I’m going to take this instance offline, it’s no longer going to be available. So my status is stopped, if I refresh this or something like that, I would actually see this not there anymore. And it’s no longer offline, the route isn’t there. So let’s go ahead now and transmit a new message. Let’s pick somebody, let’s pick our friend Andy Dwyer, and let’s say we contacted him. So the rule is none the wiser that my system is offline, it just goes ahead and triggers the rule, let’s save that. So of course it’s not going to be able to send the outbound message. Right, it’s going to fail. So let’s go prove that, let’s go back to our settings, let’s look at the Setup, we want to go to Outbound Messaging, now we want to View Message Delivery Status, so what this shows me is I’ve had 2 attempts, the next one is going to come up in a few minutes, the problem is it’s not found, it couldn’t hit the endpoint. So I can see the message Id, I can see some details about it, that’s interesting, I can see the actual object that changed, it was this, it was Andy Dwyer, and so I have some pretty cool visibility there, I could delete that saying, you know what, it doesn’t matter, it’ll be offline, that’s fine, or I could manually retry, and I can see I have 1 item in my queue. Well let’s go ahead and bring this application back online, let me start it back up again. It should now be started. Let’s go back to the page, restart it. The map is back online, so let’s go back to here and see if I refresh it’s had a number of attempts. Let’s go ahead and retry it now and see if that can make it. And sure enough, the message came across, so it was hitting errors before as the app was starting up and it was automatically retrying on its schedule. I forced a retry, and now you can see that it showed up with a new point. So I got some good reliable delivery there. I was able to queue it up, wait for the system to fail, and then manually resume it or wait for the automatic resumption to occur. So you see, outbound messaging is a nice solution for real-time integration that requires some level of reliable delivery.
So let’s summarize here. Outbound messaging is a pretty cool technology and we talked about reliably delivery, you’re dealing with inherently unreliable channels when you’re working with HTTP. So how do I work around that and build up some mechanism, typically on the sender side, to queue up things and then finally send it across, once it gets a positive acknowledge, it stops sending it. We talked about those key concepts, understanding these ideas of queuing up messages and the WSDL that you integrate and so forth. We talked about securing that either using a client certificate, of course using https for communication, other different ways. We discussed the flow for creating outbound messages and how do you create either the workflow rule first or the outbound message first and link the two together. We discussed and showed how to configure a client, you need to simply implement the WSDL that outbound messaging shows you, implement that WSDL and then we tested that locally. We showed how to track messages and actually see what happens when there’s a failure and be able to execute on that. I hope you enjoyed this module. Outbound messaging is one of the cooler technologies within Salesforce, it’s pretty wicked stuff when you’re able to send this across and see real-time behavior without polling-based things and actually see push-based notification, that’s pretty great. In the next module we’re going to see another aspect of this with the streaming API. It’s got its differences from outbound messaging, it could be attractive to you in certain situations.
Doing Push Integration Anywhere with the Force.com Streaming API
Hi there, my name is Richard Seroter, welcome to this module on doing advanced push notifications in Force.com. In the last module we looked at how outbound messaging works and how it enables some real-time notifications. This module takes that conversation a little further with a bit of a different way of doing real-time messaging with Force.com endpoints. In this particular module we’re going to start off by talking about some of the key concepts to be aware of in the Streaming API. We’ll talk about the Bayeux protocol and CometD.We’ll compare outbound messaging with the Streaming API so that you feel like you’ve got a good idea of when you should use either one or the other. We’ll discuss streaming API users and how you authenticate. Next we’ll look at Push Topics which form a foundational component of the Streaming API. We’ll discuss what it means to create clients for your Streaming API apps. We’ll look at the new featured called Generic Streaming and how you can use a capability that’s not tied directly to Salesforce objects to still do push notifications to a broad set of users. Finally, we’ll look at Streaming API limits before summarizing all of our work. What is the Streaming API? Really, it’s receiving a live stream of data changes based on a SOQL query that you define. So unlike outbound messaging where you’re really just doing some things based on the created and editing of records, here you’re actually able to have a more refined query and you’re able to define the criteria that triggers a change that you receive in the stream.
Key Concepts in the Streaming API
Let’s talk about some of the key concepts in the Streaming API. First you have topics based on SOQL queries. So these are driven, not just on simple changes, but an actual query in the Salesforce query language. So you have a little more control over defining these sort of topics and as a client you subscribe to a topic, then this becomes something you can subscribe to, this sort of channel, and poll data related to that topic, so you’re able to attach to a topic and subscribe to those changes, detach from that if you’d like to. It feels like a push API. When you see this thing, things show up instantly, it feels like a push, behind the scenes though it’s actually polling so it’s not WebSockets, it’s nothing like that. It’s actually using long polling and we’ll talk about that in a moment. Finally there’s no guaranteed delivery or ordering. Things can come out of order. If your listener is offline, your client, then you may miss some data. Now in the spring 2016 release Salesforce does add this replay capability, so you’re able to actually replay a certain window of events and collect them all, but it’s still not a foundation of your actual reliable messaging stack. This isn’t the case for that, this wouldn’t be the only way you would want to integrate data between systems.
When Do You Use It?
Given that fact then, when do you use this? Well, you’ll still use it to replace some of your custom polling-based solutions as this provides a great route for doing real-time notification of changes versus constantly polling and looking for changes and odds are, not getting them most of the time, this is a much more real-time event-driven model. The key here though is I can do these event-driven systems behind the corporate firewall because the streaming client doesn’t need to be on the internet, it simply needs to be able to access the internet.Salesforce isn’t pushing data in; the client is actually polling the data. So that gives you a little more flexibility about where to put these sort of listener applications. And then finally being able to scale these data events to multiple recipients, that sort of publish subscribe model where I could have multiple subscribers, this gives you the one route out of Salesforce that can broadcast to multiple recipients.
Bayeux Protocol and CometD
Let’s talk for a moment about what’s underneath the Streaming API. So Bayeux is this protocol for transporting asynchronous messages, typically over HTTP, and the realization of that, the implementations of that protocol, is typically called Comet, and so CometD is this event bus that implements that and it’s meant to be a simple event routing bus that does a push-like technology across the web. It takes advantage of long polling and in the modern version of CometD also supports WebSockets. Now Salesforce uses CometD 2 and so what that is is long polling. So long polling simulates the idea of a push by creating long connections up to the server and when messagesactually arrive or it matches something, it quickly can poll it down and then it opens up another connection. If nothing happens during one of its connections, it just times out and opens up another one behind the scenes. So this sort of long polling simulates the idea of a push-based, even though it’s initiating an outbound connection versus getting an inbound push. Salesforce implements the core protocol components, things like connecting and disconnecting, creating that secure handshake, and subscribing and unsubscribing from a channeler, in this case a topic.
How Does It Differ from Outbound Messaging?
So how does it differ from outbound messaging? Important to point this out here now that we know a little bit about the guts, outbound messaging is push-based. It’s sent from Salesforce to your client on the public internet. It uses a single subscriber, right, there’s a single recipient of an outbound message. These come in XML payloads. It’s based on a record or field change, and if there’s a failure it retries.Outbound messaging has that sort of store and forward, it queues the message, retries it if it can’t succeed. Streaming API looks like push, but it’s actually polling-based. I could have many subscribers to a given topic. These are JSON payloads that come across the wire. It’s based on a SOQL query, so it’s a little different, and there’s no retries. Again, you can get this sort of replay, but it’s not something where the API itself is retrying if it doesn’t hit your client, it’s just broadcasting a stream, if you happen to catch it, fantastic.
Authenticating Streaming API Users
So how do you authenticate your Streaming API users? It’s pretty straightforward. You actually can use the SOAP session ID, so that works out well. Ideally Salesforce recommends you use SOAP just for the developer testing sort of mode so that you shouldn’t use it in production, instead they recommend that you use the OAuth token when you’re dealing with production applications. As with all the other APIs, user, object, field security still applies here. So what’s key is if that subscriber doesn’t have access to the fields referenced in the query select clause and things like that, then those fields are in the notification. If the subscriber doesn’t have access to all the fields referenced in the where clause, then they won’t actually receive the notification at all. So you’re still able to reflect those user security permissions when they’re attaching to that channel or that topic, and you’re going to make sure that people don’t get data or don’t get notifications when they shouldn’t.
What Are PushTopics?
Let’s talk a little bit about Push Topics and what are they. So the record itself, the push topic record, you’ll see those in the system, relates to a CometD channel. That’s the idea. So every Push Topic record that you create corresponds to a channel in CometD. The channel name is the name of the Push Topic, prefix with a /topic, so /topic/voterTopic, things like that. Any sort of Bayeux client can then receive streamed events on that channel. The notifications are generated based on the match of a record, so when you’ve got this sort of SOQL query, the minute you save your push topic related to that, the system starts evaluating any record events, creating events, update, delete, undelete, anytime there’s a match, that new notification is generated and the server is polling for these notifications for any subscribed channels and so then the recipient will get it. The developer controls the record match criteria, they’re the ones writing whatever query makes sense, so they’re not beholden to something Salesforce figures out, you as a developer get to choose what sort of notifications you receive. We’ll look at some of those options in just a moment. You can also deactivate a Push Topic or delete it, so you could pause it by deactivating it and have any clients not be able to subscribe if you were making a bunch of changes or maybe you were doing some bulk actions or other things that you didn’t want to generate a large stream, you could actually temporarily deactivate it, no big deal. As you’re writing queries, a few things to think about. So there’s one entity per query, so very important there that when I’m writing these queries I’m only working with one entity, I can’t have a query that involves multiple ones or returns different types of objects, there’s only simply one entity per query. Any fields you’ve specified, and we’ll see what that looks like in a few moments, make up the body of the notification. So when I have a select clause, really that makes up whatever messages, whatever fields, come across on that Push Topic channel, so it’s important to figure out whether the recipient is going to receive, and I make sure that I include the right things in my query. What’s really neat is that changes take effect immediately, so as soon as I change a Push Topic query or its parameters, those changes are immediately changed on the server and all the new clients automatically get whatever is different, so they don’t have to reconnect or change their connection. This has support for both standard and custom objects, so all custom objects are supported in Push Topic queries and there’s a subset of standard objects that are supported, so most of the main ones that count, campaign, contact, opportunity task, there’s a few standard objects supported in queries as well. There’s also a basic subset of SOQL available, so the standard operators are there as well as most statements and expression and things like that, but there are some unsupported things, semi-joins, anti-joins, aggregate queries, count limit, relationships, order by, group by, compound addresses, and the like. So you clearly can’t do everything here, and you can imagine the system doesn’t want to have to do these very complex comparisons on every sort of record change. So it does narrow down the scope of what you can make the system look for, so you do have to factor that in as you’re building these queries. Let’s talk about the notification rules. So specifically, you can set up a rule that says NotifyForOperationCreate. This means that every time there’s a created record, that’s when a notification should fire. Or every time when the operation is a delete, so that’s what I care about. I can pick any or all of these, so I’m looking for delete operations that match my query. Or I can look for undelete when they’ve recovered from that. Or I can do an update. And then finally you have an option to NotifyForFields. You have for values that you can choose here. I can choose all. This means notifications are generated for all record field changes. If I have referenced, that’s the default value, then any changes to the fields referenced in either the select clause or the where clause are evaluated and sent across. Or the third option is select, that means any changes to fields referenced in the select clause are evaluated. And finally the last option is where, any changes to fields referenced in the where clause are evaluated and sent across. So you have a few different options for what sort of field should I be looking for, all of them, referenced fields, select fields, where fields, that will determine kind of which ones I’m checking to figure out, to trigger these events.
Creating a PushTopic
Let’s look at a couple of examples, a major example here, and we’ll be executing this code in a moment. So you create a PushTopic object, we’ll do this in the Developer Console. You give it a name, things like that, then you add a query. What is the query associated with the Push Topic? Then as you see all those notify parameters, if I want to pick all of those as true, then that means I’m looking for any sort of changes to the voter donation table that matches those fields. And I chose referenced fields. I’m looking for changes in the ID, the name, the amount, the donation date. If any of those change, then fantastic it will trigger one of these Push Topic notifications. If I change something that’s not in there, then it’s not going to do anything, I pick NotifyForFields Referenced, so it’s only going to change, or it’s only going to trigger something if it’s one of those four fields that changes. Finally it inserts the PushTopic record into the database, which then you can query and view like any other Salesforce database record.
Demo Description: Creating and Testing a PushTopic
Let’s jump into our first demo, in this case we’re going to create a PushTopic object, we’re going to select it to or set it to NotifyForOperationUpdate only. We’ll set it to listen on Referenced fields, so only things from my query. We’ll subscribe to it in the Workbench tool, and we’ll test it out. So we’ll see how this thing behaves by testing it in Workbench and then later on we’ll test it withinour application itself. So this should give you a good taste for what it means to work with Push Topics.
Demo: Creating and Testing a PushTopic
Here we are back in our Salesforce account and what we’re going to do here is work with the Developer Console in order to mess around with Push Topics. So Developer Console gives me a nice little area to execute some code, run some queries, things like that. So if I go to my profile, or rather the little gear, and I want to click Developer Console, this will pop up a new window that represents the Developer Console. And so where I want to go here is I want to go to Debug, then I’m going to open the Execute Anonymous Window, this lets me run some code, I don’t need any content there. This lets me execute some code, and where are we going to find this code? Well let’s go to the file system and grab that folder, if we look in folder mod6 in the code download, you’ll see a file called createpushtopicupdate. So let’s go ahead and open this, let’s copy all this, copy it to your clipboard. Let’s come back to the Developer Console and paste that in. Let’s take a look at it. So what do we have here? We’re creating a new PushTopic, I’m calling it UpdatedDonations. We’re going to select the Id, the Name, the Amount, the Donation Date, FROM Voter_Donation, API version 35, we’re not going to worry about create operation so if I create a new donation, this won’t fire. For update, yes it should fire, not for undelete, not for delete, and Referenced, so only if ID, Name, Amount__c or Donation_Date change will it actually do this. So let’s go ahead and execute this. Sure enough I see I’ve got a record down here, very good, it executed that. Let’s go ahead and prove this worked. Let’s jump to the Query Editor and let’s just run a quick query to prove that this worked. And if I run SELECT id name From PushTopic, I’m using terrible syntax here, don’t let these SQL query people see this, I can go ahead and see my ID and I can see that I have this record. So I created a new record of type PushTopic and it’s in the system.Great, so let’s say I believe that’s there. Let’s jump over to Workbench. You can get to Workbench, workbench.developerforce.com. It’s just a tool for testing against your Salesforce environment. I’m going to go ahead and login. I will allow access to this account, and then once I’m in here I’m able to mess around with the environment here. I can do queries, I can do search, but I want to use queries Streaming Push Topics. This is great because I can pick a Push Topic, there’s my updated donations, and I can subscribe. So now I’m subscribed. I connected to it, but now I’ve actually subscribed to that and I’m subscribed to topic/UpdatedDonations, fantastic, I’ve proved it’s connected. And so now this app is actually long polling for me right now. Let’s go ahead and go to Salesforce. Let’s go to Voter Donations, I’ll get to it from my app launcher. I’ll pick Sales. Let’s go to Voter Donations. What I want to do is edit the candidate’s name. So if I come here, if you remember, candidate name is not part of my Push Topic query, so let’s just prove that that’s not going to fire here. We’ll say it’s Mike Smith Jr who’s actually running, he’s riding his father’s coat tails. We’ve saved that. Let’s go back to Workbench. Nothing. Nothing happened here because I’m not subscribing on, remember I picked Referenced as a value, which means it wasn’t in my query, therefore I’m not going to get it. So let’s go ahead and while we’re still keeping this open, let’s go back to our developer console, okay we’re in our developer console, let’s go back to this and let’s delete all this and what I want you to do is to go to your file system again and I want you to open this editpushtopic. Let’s copy that, save that, come back here and paste that in. So what we’re doing here is we’re pulling the list of Push Topics. To edit them I’ve got to go ahead and pull a list of them, I’m going to select the Push Topic where the Name equals UpdatedDonations, that’s the name of the one we’ve built. I’m going to go ahead and instead of Referenced, I’m going to change it to All, this means any change will go ahead and trigger this Push Topic, even if it’s not referenced in the queries. So let’s go ahead and then update that database. Let’s execute that. Alright so it’s gone ahead and successfully updated that. So now let’s go back to Salesforce and let’s go ahead and change the name again. Now it’s Mike Smith Sr, the grand statesman has decided to run for office here, and sure enough look what we got. We got a new update, amount 1000, Id, Name, Donation for, that’s this record, so again, it didn’t pull back the candidate name because that wasn’t in my query, but because we updated the candidate name and our channel or our topic was now looking at any field, all of them, not just referenced ones, it went ahead and kick started this, that’s pretty cool. So to clean these things up I have a few options, if I wanted to get rid of my Push Topics, I could go ahead and go back to Workbench and issue a query to actually delete this particular record from the Push Topic. I could do it from the database. Or I can also decide to come into here and if I look at the details, I can delete it from here. So I have a few different places I could choose to clean up after myself, I could do it in a query, I could jump into the anonymous window and execute some code calling it database.delete on the Push Topic I retrieved from a query, or I could come here and do it from the Workbench. So I have a few ways to clean up after myself. So what we saw here was creating a Push Topic, setting the field initially to Referenced and then seeing that we didn’t actually get it when we subscribed to it, switching it to All and then seeing the change to a field not referenced in my query in any way still triggered the message received to the topic in real-time.
Creating Streaming Clients
In that last exercise we kind of used a fake streaming client using the Workbench application to receive the streaming event, so what if you really want to build an actual client that receives these streaming events? Well you have a few things you can deal with here. First of all, my first piece of advice is find client libraries. Look for CometD libraries, Node.js has one, we’re going to be using it here, but you want to save yourself from creating all this handshake, reconnect sort of logic. You’d like to use a library, whether you’re using Java or .NET or Ruby or whatever. Find something that’s going to play nice with the Streaming API with Salesforce or look for something that uses CometD. Next you have to think about your connection to the channel. You have to make sure that your client can work with the Streaming API, can respect the cookie protocol that comes with this, it has to be able to accept and send the right cookies to the domain and URI paths. So you just want to make sure that it implements some of the core requirements of the Streaming API, kind of deal with content type of application JSON, kind of make sure it passes in all the session ID information. Can it send back and accept all the necessary cookies that come from the server? Then of course you have to handle the part of subscribing to messages and getting the messages in, you’re going to be getting these messages coming in in JSON and your client is going to have to process that JSON and do something with it. Now unlike outbound messaging, you don’t have to send back an Ack, you don’t have to send back this acknowledgment that says I received it, the streaming app doesn’t really care, it’s just pumping these out there and if you get it great, if you don’t it also doesn’t care. And then if there’s time outs, again, if you use a good library then it should handle a timeout and just immediately open up another connection to the server. If you’re somehow writing this on your own, you’d want to make sure you explicitly handle these timeout scenarios or else you’re going to constantly miss data points.
Demo Description: Configuring a Streaming API Node.js Client
Let’s do a demo where we actually implement this in an application. So we’re going to now review this donation streaming application that I’ve built that we’re going to mess with. We’re going to update the user credentials to use your Salesforce credentials, we’re going to go ahead and set the client subscription and what we’re looking for, and then we’re going to test that with the Force.com data changes so you can really see the streaming client in action.
Demo: Configuring a Streaming API Node.js Client
All About Generic Streaming
Let’s talk a little bit about generic streaming, this is a fairly new capability in Salesforce, and this really the point here is it gives you the idea to send notifications on general events, things that you describe, it’s not tied to an object, but you’re using Salesforce.com as a broadcast mechanism for general purpose. The streaming channel object is just a regular creatable Salesforce object, so you can create one programmatically using Apex or any of the APIs, like SOAP or REST. You send events through the REST endpoint, so you use the streaming channel push REST API resource to generate event notifications to any of the channel subscribers. So imagine having a bunch of mobile apps subscribed or a bunch of apps subscribed, again, you could just use this as a general broadcast, it can have nothing to do with any record changes, just a generic capability for doing streaming. What’s also neat is you can target a subset of subscribers, so you can specify the list of subscribe users to send that notification to instead of broadcasting to everybody. So you can use a GET method on that channel push REST API to get a list of active subscribers, and then you might want to down select that based on some criteria you have that only this set of recipients should receive this or this set of recipients should receive that. So it’s pretty neat, a little bit of sophistication there, it’s not a crazy service, but it’s a cool way to build on and piggy back the basic streaming capabilities offered by Salesforce.
Demo Description: Using Generic Streaming
So let’s do a quick demo with this. We’re going to go ahead and create a generic streaming channel. We’re then going to define a subscriber in our client app, kind of hijack that to demonstrate generic streaming. We’re going to send messages to the REST endpoint using Postman, and then we’re going to observe the output, just do a quick demo and see how this capability works and see how simple it is to use it.
Demo: Using Generic Streaming
Alright let’s go ahead and create that streaming channel, the generic streaming channel. Now I can’t seem to find a way to do this in the Lighting experience just yet, so we’re going to go ahead and switch to Salesforce Classic and from this homepage you hit the plus over here at the top, and what you’ll see down here is the option for streaming channels. Pick Streaming Channels. We don’t have any streaming channels here yet, that’s fine. Let’s click New, and as you can see, there’s not a whole lot to it. I can give it a name, so in this case let’s go ahead and do something extremely uncreative, it does require this sort of prefix, so I’m going to do u/notifications/DemoGenericChannel, that’s a horrible name, if you want to make something better I will not limit that at all. So go ahead and pick a name, but do have this prefix, a /u/notifications, that is important. Then go ahead and just click Save. Great, so now I’ve got a generic channel. There’s not much more I need to do there. So let’s go into the Developer Console, and I’m going to go ahead and switch back to the Lightning experience before I forget, and while I’m here what I want to do is I want to jump into the Developer Console, I’m going to jump down to the Query Editor, and here what I want to do is I want to select Name and ID FROM StreamingChannel because I need to get the ID of that channel I just created. Here it is. So I can see that value. I’m going to copy that because I’m going not need that later. And so let’s go ahead and jump to Visual Studio Code. So here I’m in Visual Studio Code, you could be using any IDE you want, again it doesn’t really matter, let’s go ahead to where I’ve got the topic subscription in the app.js file. Here I’m subscribing today on AllDonations. Let’s change that. Let’s instead subscribe on u/notifications/DemoGenericChannel. So now I’m subscribing to the generic streaming channel, great. I want to comment out these sort of things, I’ve already got those commented out. I’m going to change this one because I’m no longer getting an amount and I have this bottom one I left in there for streaming messages. Here I’m just printing out the payload and I’m not sending any amount over, of course it doesn’t apply, we’re really just hijacking our previous app. And in this case I just want to throw out whatever I’m sending. So whatever is in that payload, send out to the client application, does that make sense? So I’m going to click Save and I’m good to go. So let’s go ahead and start this application. We’ll launch it. Great, I connected successfully. I’m connected to the generic channel now, that’s great. Let’s go ahead and jump into our browser. I’m hitting localhost:3000 and once again I’m connected. So now let’s go ahead and send a message to this. I’m going to jump into Postman and from within Postman what I want to do is a couple of things. So I’m going to create a new POST request because we want to send the message across and this is going to be https:/na30 or whatever your server name is, make sure you remember to use whatever the right one is that comes back for your account, services/data/v35/sobjects/StreamingChannel because I’m using the streaming channel and it’s really a REST sort of call here. Here is where that name comes in handy. So here’s that ID that I pulled out of the Developer Console, I need the ID of my streaming channel so I know which one to push to, and the /push. So make sure you’re grabbing this from the Developer Console. Next what I need to actually need to be using the REST Bearer token, I want to go ahead and use proper OAuth authentication here, so I’m going to create new tab and refer to the REST login that we’ve done before. Remember I’m sending this request to login.salesforce.com/services/oauth2, passing in all the values like client_Id, secret, username, and password. And what I want to get back from this is this access token. So I’m going to take this token because I need this for the streaming client, I’m going to come over here and in the header add an Authorization header, this will be for Bearer and I’ll put that value in, so great, so I’m authenticating myself. Then what I want to do is what’s the body I want to send across here? Well it’s going to be a JSON body, so let’s set it to application JSON, and now let’s jump to our file system and you’ve got something here called genericstreamingrequest. Open this up and you can see a remarkably unoriginal payload, let’s copy that, paste that in, you can see I could specify userId, so this would be users that would be allowed to receive this message. So let’s say the payload is New voter pool identified. In this case I’m specifying this format that’s expected by the streaming channel, the generic channel expectsthis payload and sort of thing. Let’s click Send. Let’s make sure we have a good URL here, that helps. Click Send. We get back fanoutCount. Alright so let’s go ahead and look at our client application. Sure enough, New voter pool identified, so it went ahead and hit our streaming channel, so we did generic streaming, let’s go back again and just for fun make sure that this is actually working. Alright we’re going to send a new message that declares our love for Pluralsight, click Send, let’s go back, and sure enough it sends it. So I’m using generic streaming, it has nothing to do with SObjects, has nothing to do with changes, but I’m able to use the same streaming mechanism and the same streaming engine to send all sorts of messages. You could imagine a lot of different uses of this, that’s pretty handy.
Streaming API Limits
There are some limits enforced by the Streaming API, as you can imagine. There are 50 topics allowed per organization by default, up to 100 depending on your account type, but typically you see around 50 for enterprise. The maximum number of client subscribers per topic is 1000 for enterprise, up to 2000 if you have the performance or unlimited account. And so again, a pretty good number of clients, but this is probably not enough to use out with your entire force of sales people all connected via mobile devices, so it’s not meant for that sort of distribution scale. The maximum number of events per day, any sort of 24 hour period, is 200,000, it’s a million, again, if you’ve got the performance or unlimited account type, but still that’s a lot of good events and hopefully good shape there, and then you can do up to 10,000 maximum generic streaming events per day. So still, again, good volumes, but be aware of these limits because this is going to factor into potentially how you use this.
I hope you enjoyed this module. The Streaming API is a lot of fun. We did an overview to kick things off. We talked about some of the key concepts around Push Topics and otherwise. We talked about the underlying protocol and how this works and making sure that you recognize there is something underneath there, it may not be using the most modern version of CometD where you have things like WebSockets available, but you have a very reliable long polling sort of mechanism that can work from anywhere, behind your firewall and likewise. We compared outbound messaging and the Streaming API, hopefully you saw some clear differentiators around one-to-one versus one-to-many, or exposed to the firewall or not, or using SOQL queries versus just data change events. A lot of different things there, and so hopefully you do see there’s some similarities, of course, but that there are some real differences that can help your decision make process. We talked about authentication using either the SOAP API or, like we just did in our demo, with the REST Bearer token with OAuth. We covered a lot about Push Topics and just took a look at how you create them, what the essence of them is, what sort of things you deal with on the query and what events you listen in on and what sort of field values you’re actually looking for changes on. A lot of different things to consider there. We talked about creating Streaming clients. We discussed Generic Streaming and how you could use this of general purpose broadcast. And finally we discussed some API limits. Streaming API is definitely a lot of fun. I hope that you actually try to use this a little bit more as it’s something that can save you from polling and be pretty easy to set up and easy to maintain. In our last module coming up, we’re going to take a look at Apex callouts.
Consuming External Services with Force.com Apex Callouts
Hi there, my name is Richard Seroter. Welcome to this final module in a course about the Force.com integration APIs. In this module we’re going to talk about consuming services outside of Force.com. While in the last module we looked at the streaming API and we’ve looked at a lot of ways that you can kind of reach data into Salesforce or Salesforce pushes some data out, what about ways for Salesforce to reach into other systems and pull in data to make it easier to integrate with these external systems? That’s what we’ll be spending time on here, how does Salesforce dip into other systems and pull in data. We’ll start off by talking about some of the key concepts in Apex callouts. We’ll discuss remote site settings and what they do. We’ll share the idea of named credentials, which is a fairly new capability, talk about how you can generate Apex classes from a WSDL, perform raw HTTP callouts if you want to get down to the bare bones and work with HTTP objects and JSON and XML. We’ll see how you work with XML and JSON payloads. We’ll take a look at long-running callouts from VisualForce pages, which is some pretty cool stuff, and then finally we’ll talk about some of the callout limits and what you have to be careful of when you’re calling out to external systems. So what is an Apex callout? Really it’s making a call from Apex code, Apex being the programming language and framework within Salesforce, to an external web service, and then receiving a response and doing something with it. Again, pretty straightforward, but important to understand what these terms are and try to give you some basic definitions.
Key Concepts in Apex Callouts
Let’s talk about some of the key concepts here that you should be aware of when working with Apex callouts. You can either use generated classes or raw HTTP objects. So you might like working with generated stubs, if you’re targeting a SOAP endpoint you can do that pretty easily and we’ll do that here together. Or sometimes you want to work with the raw HTTP objects and the payload because you want to have maybe a special request with custom headers or some other thing where you need to have lowest level access to thecommunication and you don’t want the abstraction provided by a generated stub. What’s neat is with these named credentials is now you can more easily decouple credentials from code. You don’t want to necessarily have to embed the URL and credentials in your code or even have to create kind of fake extra objects to store credentials and then manage those, instead you’re going to have a first class citizen sort of solution for managing credentials and inserting them into your application at runtime. You can do asynchronous processing, both from stubs as well as within VisualForce and other things so we don’t always have to do synchronous processing, which is great. And you do have to be aware that typically you’re not allowed to do these callouts after DML operations in the same transaction, so by default callouts aren’t allowed there in the same transaction because DML operations result in pending uncommitted work, and so you can’t have callouts right after that. So you just have to be careful where you put these in your code, you don’t want to do it after these sort of, or within database transactions because otherwise it’s simply not going to work with the way that Salesforce does transaction management.
When Do You Use It?
So when do you use this? Well a great use case is simply aggregating Salesforce and external data in a mash up. I want to make sure that I’m taking data from Salesforce, I’m taking data from external systems, and instead of copying data into Salesforce all the time, how about I access that data where it lies. And instead of having to pull it in and store it and sync it, let’s go ahead and get a live look into these other systems. Your other options include creating VisualForce pages based on data from other systems, which we’re actually going to do as an example. So again, instead of trying to copy that data in, I can use these pages and actually pull the data in live from some other system.Or finally doing some sort of long-running asynchronous queries, again, it could make sense to be doing from code and doing things asynchronously that process later.
About Remote Site Settings
Let’s talk about remote site settings. So what this is is it’s a way to register to authorize external sites. Before you can access any external server from an endpoint in Apex, or any other place, you have to add the remote site to a list of authorized remote sites that Salesforce knows about. Make sure that your code doesn’t start doing things it shouldn’t or starts phoning home to places it shouldn’t be able to, so you have to explicitly tell Salesforce what you’re allowed to callout to. And then you have some choices about HTTPS and some other things, it gives you a much more explicit way to reach out of Salesforce and do that with your eyes wide open, not just having code that starts writing all over the place to different places. You provided a name and a URL. You can choose to do SSL or not with that endpoint. And you can even make these things active or inactive so, again, you may temporarily make something inactive while you’re getting ready to roll it out. You could use this for a variety of reasons, but remote site settings are a pretty key part of being able to do callouts.
Demo Description: Creating and Configuring a Remote Site
So we’ll do that here, we’re going to take a look at a Node.js app that creates, that returns rather, the positions that a candidate has. Let’s say that a typical candidate changes their positions all the time so we need a nice flexible external app that returns it live, that Salesforce users can view it within Salesforce at any time. We’re going to deploy that app, we’re going to test that app, I’ve actually added some capabilities for basic authentication so we can also test that. We’re going to locate the remote site settings within Force.com and add an entry for the positions app URL.
Demo: Creating and Configuring a Remote Site
Alright in this exercise what we’re going to do is deploy our positions application, it shows the candidate positions. We’re going to go ahead and push that out, and then we’re actually going to set this up in Salesforce with its remote site settings. So here I am in Visual Studio Code. I opened up the Positions application, if you go to your File, Open Folder, it’s the PositionsApp in the mod7 folder. And so this has a few things in there. If you remember, we first want to make sure we install all the dependencies located in the package.json. So you’ll want to go to your command prompt in your PositionsApp and do an npm install. What this does is load all those dependencies so that we can do local testing. If you were pushing this entirely to some fabric that did it for you, you wouldn’t have to care about that, but we like local testing. We can prove that worked by seeing the node modules folder now show up. So this is a pretty basic service. All we want to do here is, if you look at the position.js, this has some basicAuth capabilities because I want to do some basicAuth and show how we use named credentials in a moment. But basically there’s two endpoints, there’s the base one that does a basicAuth check and it hits that at the base URL and returns an array of candidate positions, and then we have a /async that also returns that positions array, it doesn’t worry about basicAuth, but it does have a 5 second time out, so this is one we’ll use for asynchronous processing later because it takes longer to return. So we’ve got this working. Let’s go ahead and just prove that this works by testing it locally. Let’s switch to the little debug experience and run that, if it’s the first time you’ll have to pick Node.js to build this little manifest, and then start it up. If we jump into Postman we can test this by hitting localhost:3000/positions. Now when I do this first, remember the /positions expects a basicAuth header. So if I go back up to the top here, switch this to basicAuth, add one of Pluralsight and the password is testing or password, rather, let’s update the request, let’s send that across, and sure enough I get my array of positions. Now if I do this /async, this one does not require the authorization header, so let’s turn off authorization here and then go to the Headers and make sure we remove it. Here it just takes 5 seconds to return. So this will be one you’d want to call asynchronously because it takes a little while, but we get everything back.So we’ve proved our service works, now pick your provider of choice to run this. I’m using CenturyLink’s AppFog, you could put this in Heroku or Amazon or Azure or any sort of location, just get this up to the public internet. So for me, I’m just going to do a simple cf push of my positionsapp-seroter as the name of it, it’s a node 4 something application so I’m going to use a build pack that’s friendly to node.version 4. It’s going to update and now I’m going to have this on the internet, meaning that then an Apex callout can reach into it. After a few seconds my application is online. We can prove that this works by switching the URL from being localhost:3000 to being my AppFog URL, so this would be positionsapp-seroter.uswest-appfog.ctl.io, and I can hit the async one, for example, this should still take 5 seconds, let’s make sure I’m actually hitting the right URL here. It always helps when you type in a proper URL format. Let’s hit that with async, hit Send, this should take about 5 seconds while it does its usual delay in this web service call, and sure enough I get back my positions. And just like before, if I hit the straight up positions URL it’s not going to like it until I pass in a header, which we’ll use in a moment later on. So let’s go ahead and make sure we have this URL. I’m going to copy that. Let’s go back to Salesforce. And from here what we want to do is set up a remote site setting. So go do a search for remote, go to Remote Site Settings and we want to add one specifically for this endpoint so that we can call out to it. So we’ll call this PositionsApp, set that remote site, we’ll keep the https on there so we don’t want to disable security, and save that. So now we’d be able to call out to this endpoint with no problem. We looked at our app, we deployed it, we tested it, and now we’re ready to consume it in an Apex callout in an upcoming exercise.
Using Named Credentials
We just set up the basics of our application so we get ready for an Apex callout. Let’s talk about named credentials. We just deployed an app that uses basicAuth, so if I’m calling this in my code, right now I would have to apply credentials to that and that can be risky as credentials change or I don’t want that embedded in my code where anybody could see it. So named credentials is a fairly new feature that lets you separate the code from the URL and the authentication. So the named credentials specifies the URL of the callout endpoint and any required authentication parameters, here’s the password or the OAuth token or things like that. Technically you can skip the remote site settings if you’re also using named credentials. You don’t have to do the same together, you can do it both to be safe, but this actually acts as the same thing, it’s letting you securely call an endpoint. This is also handy if you want to use the same code for different environments, so by separating the endpoint URL and authentication from your code, it makes these callout simpler to maintain because if the endpoint URL changes, I just update the named credential, I don’t have to go update or deploy any code. If I’m moving between environments from dev test prod, there’s ways, again, I could simply update that URL as it’s moving between stages without ever having to change my code. My code can stay universally the same either in source control or even when it’s deployed, and I can just muck around with the named credential only. I can call things anonymously if there’s no security necessary. I can use basic security, really password security, and I can use OAuth with Bearer tokens. So I have three different options I can use with named credentials when I set that up.What I really like too though is that you can do merge fields. So let’s say you don’t want Salesforce automatically injecting this authentication header, maybe your web service has a different security scheme or something where it can’t use that header. Instead what you can do is actually access merge fields. So I can access the credential username, password, OAuth token, authorization method. I can access all of those things in code with tokens so that it pulls the value from the named credential without actually kind of automatically injecting it. So I could build up a custom header, I could put something into a SOAP payload, whatever I want, I can use those tokens however I want to represent the actual values. So pretty cool that I can use merge fields if I want to do a custom authentication or authorization scheme.
Demo Description: Configuring a Named Credential
In this brief demo what we’re going to do is we’re going to find the named credential settings, we’re going to create an entry with password authentication, we’re going to take a look at just a few of the options available as well, just so you get a good sense of how named credentials work.
Demo: Configuring a Named Credential
Alright so now let’s go ahead and set up a named credential for our endpoint. Remember that our non-asynchronous endpoint does basicAuth, so it’s actually looking for credentials before it allows you to call it. So let’s go ahead in our setup screens in Salesforce and look for named credentials. I want to add a new one here, let’s call it PositionsCredentials. The URL is that positionsapp URL. We have a few choices, we can deal with certifications, we can do anonymous, we can have it be a per user thing with a password applied specifically for an admin or obviously using OAuth here. We can also make it a named principle, which also has an authorization provider in scope. In this case, we’re going to use Password Auth for the named principle and the username is Pluralsight, the password is password. And if you look down at callouts, you can tell it to automatically generate the authorization header, allow those merge fields, so you can take advantage of building up a custom header if you’d like or a custom payload. I’m going to go ahead and save this and now we have a named credential that we can use within our code instead of directly embedding the code itself or the URL itself or the password itself in my code.
Generating Apex Classes from a WSDL
Let’s talk about how you generate Apex classes from a WSDL. So before we start consuming these services with raw code, let’s see how you can do this even simpler if you’re dealing with a SOAP service. So it uses a WSDL on the local drive. If I have a WSDL for an external service, right, again, this still has to be an internet accessible web service, if it’s a SOAP service with a web service definition I can save that WSDL to my local machine and then it generates a default class, I can change that name if I want to, but it generates a class with all the different typed objects that represent that endpoint. It generates both synchronous and asynchronous classes, so I can call this from code in a synchronous or asynchronous fashion, so that’s great. I’m invoking it then using these strongly typed classes. I’m never dipping down into the details of a SOAP call, I’m simply creating instances of these generated objects, calling the operation, and behind the scenes Salesforce is taking care of all the serialization and invoking the endpoint and things like that. You can work with input and output headers, so if I want to add some headers to the request, I’m doing compression, I’m doing something else, I have access to that and I can access output headers as well. It supports most of the data types and WSDL constructs, so as you’re doing certain things, if I have WSDLs with multiple port types that’s not supported, or I can’t do multiple services or multiple bindings and I can’t handle importing external schema, so the schema would have to be embedded within the WSDL versus pointing to fragments and a bunch of import statements. So there are some constraints, again, but it does a pretty nice job taking what it can from the WSDL and making it easy to invoke it.
Demo Description: Generating and Testing SOAP Stubs
What we’re going to do now is locate Apex classes in Force.com. We’re going generate a class from a provided WSDL, I’m going to point you to just a public internet accessible SOAP service, and we’ll generate a WSDL from it. We’ll go ahead and call that service from the Developer Console and just see how it works. So just show that I can easily consume an external service and then invoke it from code with no problem.
Demo: Generating and Testing SOAP Stubs
Alright so let’s consume an external web service and generate classes in Salesforce, invoke those classes, and see how easy it is to call an external web service. So in the setup screens what I want to do is I want to go to Apex Classes under Custom Code, click Apex Classes and you’ll see an option to generate from a WSDL, so let’s find a WSDL. So it’s not always easy to find internet accessible SOAP services that don’t require passwords or registration, but I did find one. So if you go to service-repository.com/service/overview and the numbers877027757, this is a simple calculator web service. It’s a simple SOAP service, not much to it, but it will serve our purposes. So go ahead and download this WSDL file. You can save the link, you can save this to your mod7 folder, and there you go. So now I have my calculator WSDL, that’s great. Now jump back into your Salesforce experience, let’s generate from a WSDL. You’ll notice this only takes a file, so that’s why I had to save it locally. I can pick that calculator WSDL, click Parse and it parsed it successfully. Then it gives it a generated name, that seems super easy to remember, so let’s go ahead and keep that, it just generates it from some data within the actual WSDL, it should be easy to find. Let’s click Generate Apex Code and it goes and does that. So it generates a couple classes, an async version and a traditional version that’s going to be a synchronous operation. I can see the class it generated, this is just typical Apex code. Within my code I’m going to be using this iCalculator object. I can call things like add, subtract, others, so here’s an add method, here’s a multiply method, things like that. Simple math. But it’s actually going to be invoking this sort of endpoint, right, so it’s going to be calling out, but I don’t want to have to even deal with that in my code. Let’s make it super simple. So let’s go to the Developer Console. Let’s go ahead and open the Execute Anonymous Window, get rid of the PushTopic from the last time we were here, and now let’s execute some simple code. What I want to do is I want to go ahead and consume that service. Now if I try to consume this right now, let’s do it first and see what happens, and then hopefully you can uncover what the problem is going to be. So I’m going to use this object, if I hit Ctrl+space, I do get some IntelliSense sort of behavior here. I want the iCalculator class and we’ll call that c, it equals a new instance of this .iCalculator. So I create a new object, Double d, we’ll do c.add and we’ll add 10 and 20 together. Great, and let’s go ahead and export that out. We’ll do a system debug statement and say value is d. We’ll do that Double d = c.add, we’re going to add 10 and 20 together, and then print that out in a debug statement. Let’s execute that. Oh no, it doesn’t like it. Guess what? Because I didn’t actually have remote site setting to this obscure URL location, so that’s no good. So we have to make sure we go ahead and add this endpoint as a remote site setting because Salesforce is saying you can’t call out to that because I don’t know what this is. So that’s great, let’s leave this open, go back to Salesforce.Let’s go back to Remote Site Settings and we need to add an entry for this. So let’s go ahead and click New Remote Site, let’s call this CalculatorSite, put in that URL, we’ll disable protocol security because it’s just an HTTP site, click Save. Let’s go back to the Developer Console, let’s execute this again, open up the logs, and I can see some values. Let’s go ahead and see the output of this. Filter down the debug statements, and sure enough, value is 30. It was able to add 10 and 20, got back 30, if that’s the part that impressed you then I’m a little worried about you, what should have impressed you is that this call to SOAP endpoint from our code, we didn’t have to write anyactual SOAP or HTTP code, it used our proxy object to invoke an external web service that was secured because I entered it in the remote site settings and it trusted it. So I was able to execute that, check out the log, filter that down to just the debug statement that we added in code, and prove that we were able to consume an external service with a generated stub.
Performing Raw HTTP Callouts
So we just walked through this idea of using generated code to a SOAP service and calling that code and being able to invoke an external web service. That’s convenient, but oftentimes not realistic because you actually need to go a level deeper and have more control over the actual web service request. There may be some specialized things in the headers, there may be some formatting that you want to do on what’s going in and out, you just may simply want more control. So what I want to do raw HTTP callouts, you have a number of choices here, there’s a number of classes that Salesforce provides that let you get down in the weeds. There’s an HTTP class. This lets you initiate an HTTP request and response, has a send operation that then takes in that request and returns a response. There’s an HttpRequest object. This lets you programmatically create HTTP requests like get, post, put, delete, and then you’re using XML and JSON classes that we’ll talk about in a moment to parse XML or JSON content in the body of a request. Conversely you have an HttpResponse. This handles the HttpResponse returned by that HTTP class and then again you use XML or JSON classes to parse the XML or JSON content in the body of a response. And this has a number of methods on it like getBodyDocument, getHeader and so forth. The key is that both of these are fairly rich objects for getting content and calling HTTP. You can use named credentials here, so instead of an endpoint URL and saying hey set the endpoint to this specific URL or add these credentials, you can actually set the endpoint to a callout, and then you’re notembedding anything in there except the reference to a callout or a named credential, rather, that you can change anytime without having to change your code, and you can use the @future designation on an operation to run async. So the previous example I just showed you ran synchronously, meaning that no further processing can really happen until that web service comes back. You can use the @future annotation to make these callouts run asynchronously and that works fine.
Working with XML and JSON Payloads
Let’s say then you’re working with XML or JSON payloads, how are you supposed to do that? Well if you’re working with JSON you have a number of objects here and so what’s nice is in the last little while Salesforce has increased the supportability for JSON so that you can serialize and de-serialize objects, SObjects, and all sorts of things fairly easily. So even your Apex classes on top of your SObjects, whether they’re standard or custom, can all be serialized fine with JSON. So there’s a system.JSON namespace instead of objects that has methods for serializing Apex objects into a JSON format and then de-serializing stuff that comes back. There’s also a JSON generator class that has methods used to serialize objects into JSON using the standard sort of JSON encoding. You could actually build up the object this way, writeDateTime, writeString, writeBoolean, and you can build up a JSON message using this. Likewise there’s a JSONParser class thatrepresents a parser for JSON encoded contents. So how do I process through this and get values at specific positions? XML support exists via an XMLStreamReader, so here are methods for forward ready only access to XML data. It’s not just a DOM object, instead it’s a forward only stream through that data, and you can pull data from XML or skip unwanted events, things like that, you can parse even nested XML up to 50 nodes deep, so it’s a pretty good StreamReader. As you can imagine, you also have a StreamWriter, so this is programmaticallyconstructing an XML document and then sending that over to an external server, so I can build this up in a streaming fashion as well. If you’d like to work in a sort of DOM model where I’m able to go up and back and I’m able to navigate the tree without dealing with a stream, I can use the DOM document objects. This lets me process XML very simply, very easy to work with, it’s not going to be as performant as a stream, but it’s a lot more convenient. When I’m dealing with Dom.Document I’m typically working then with Dom.XmlNode, this actually lets me work with an individual node in an XML document.
Demo Description: Making an Apex Callout from a VisualForce Page
So let’s do a demo here, what we’re going to do is we’re going to create a VisualForce page to show the candidate’s position. Salesforce users want to look at a live look at what the positions are for their candidate. We’re going to add a custom controller, and I’ll give you all the code for this, so really I want you to focus on the exercise, not worrying about coding. Then we’re going to call this position service that we have running wherever you’ve got it running and we’re going to parse the JSON that comes back from that. And then finally we’re going to go ahead and test the page.
Demo: Making an Apex Callout from a VisualForce Page
We’re back here in the setup screens and what we want to do is create a new VisualForce page and controller, test out these named credentials, and see how all this works within a page. Let’s start off by going to Apex Classes and let’s build our controller first. Click New, we’ll generate a new class, and then jump back into the file system where our mod7 files are. What you want is the PositionsController, just copy everything in here and we’ll look through it here in just a moment. Take that, paste that into the Apex class and let’s take a look at what’s in here. We are creating a class called PositionsController, we’ve got some, a getter, if you will, to get back the PositionsList, and then we have an operation called GetCandidatePositions. We instantiate the HTTP object, instantiate a request, set the endpoint to a callout. Let me go ahead and duplicate the tab, make sure I name this the right thing. Let’s see what the name of our named credential was, I clearly have no memory left, so let’s see, we want to search for Named Credentials, and it was PositionsCredentials. So let’s make sure we’re calling that out correctly. This is PositionsCredentials. And then I’m appending anything else that’s part of the URL, in this case positions. So this is really a token that references the callout instead of embedding this code directly, which is great because if I just embedded this code, which we’ll try later, I don’t have the appropriate header to do basicAuth. So in this case by using this one, it’s going to use the credentials embedded in the named credential and it’ll pass in that password. I’m doing a GET. I’m creating a response object, which comes back from sending the request through the HTTP object. I’m getting the body, I’m doing a JSON deserialize the payload, get it all back into a map of a string and an object. I’m looping through all the positions and then building up an HTML string that I’m going to show on the page. I’m going to save this and then I’m going to jump down to VisualForce pages and let’s build a new VisualForce page.Let’s click New and then let’s jump over to our file system and you have positionspage, let’s copy this whole thing, let’s paste that in here at the bottom and we’ll call this PositionsPage, and let’s scroll down and take a look at that. It’s very straightforward, it uses our custom positions controller, has a little custom style sheet here. Then it has a button to get the latest positions, which then calls that action. Again, this action then reaches out to our external service, loads the response, and then we put HTML directly into this output text field. So let’s go ahead and save this, and then let’s preview it. So if this works correctly, it should have authenticated the user using basicAuth and returned all of the different positions. That’s great. So that showed that it worked correctly, this is pulling directly from that service hosted somewhere outside of Salesforce. Let’s see something else though. Let’s go ahead into our controller, so let’s go back into the Apex Classes. I mentioned that if you take a look at this and go into our class, and let’s go ahead into this controller and edit it, if I happen to go and swap out the callout for the direct endpoint, this is still valid, but I can still make this call. Where this is going to struggle is I didn’t have the password in here because instead of using the named credential I would have to have done this myself. So if we go back to the VisualForce page and now try to run this, we’re going to see there’s an issue here. So let’s go ahead and preview this and this should fail because I’m not going to authenticate the person. Sure enough, it’s erroring out. It’s very confused, doesn’t know what’s happening because it didn’t pass in the right value. So I’m able to use the named credential successfully, which saves me from having to worry aboutgoing in and actually setting up basicAuth in my code and embedding passwords and any of that sort of stuff. So much, much simpler to be able to come in instead and edit our controller back to instead be using the named credential, which takes care of all that basicAuth, takes care of my URL, and I don’t have to worry about having a broken application. So in this exercise we created a new controller, created a new page, showed that we were able to call a service outside of Salesforce and have it in the Salesforce experience.
Long Running Callouts in VisualForce
Let’s talk about long running callouts. So when I want to do these sort of callouts out, that’s great, I can do that in VisualForce. And why do I want to do that? Well users feel like things happen faster, a typical Salesforce app is going to benefit from these asynchronous callouts when there’s a button the user clicks and they’re waiting for data. Instead of having them sit there and wait while the screen is frozen, VisualForce pages can continue to render and then wait for a callback. How does this actually work? Well the idea is that there’s a continuation, that’s an asynchronous callout. So these callouts actually go through a continuation server, which coordinates the call to the external service, when it gets the response back it calls back to the VisualForce controller, which can then respond and render something on the screen. So it does this sort of chaining activity to make sure that the VisualForcer user doesn’t really see all the complexity behind the scenes. What’s pretty neat is you can either do these calls in parallel, so if I made multiple long-running calls from my controller I could do these in parallel so they all run out there and then when they all come back I can write back to the VisualForce page, or I can actuallychain them if the order matters and if the callout of one is conditional on the response of another, I can chain these requests so that the callout only happens when the first one finishes. A lot of options there for you to do, makes it a little bit easier to build more responsive systems.
Demo Description: Creating Long Running Callouts in VisualForce
Here in our final demo what we’re going to do is create a brand new VisualForce page and controller, and this one’s going to be set up to use asynchronous and continuations instead. Now this is going to be using the no authentication endpoint in our service because you actually can’t use named credential along with this sort of asynchronous callout. The continuation server doesn’t know what to do with the named credential, it doesn’t work. So it may work in the future, it will probably come on an upcoming release, but for right now you can’t use named credentials with these sort of asynchronous callouts. Finally we’ll test the page and just see how things work in an asynchronous fashion using our web service, if you remember, that has a purposeful 5 second delay, so we’ll feel that pain just a little bit.
Demo: Creating Long Running Callouts in VisualForce
If you remember, our positions app has an asynchronous endpoint with no authentication that handled a 5 second delay and it returns, it takes 5 seconds to return. So let’s go ahead and build a brand new Apex controller that’s going to do an asynchronous call to that endpoint instead of making the poor VisualForce user wait the 5 seconds to get back the positions. Let’s create a brand new class and let’s jump back into our file system, wherever you’ve dropped the mod7 file folder. Now let’s find positioncontinuationcontroller, let’s save this and we’ll go through it in just a moment. Let’s paste that in here and let’s see how it’s a little bit different. Now I have a continuation ID that I want to store here at the top as a variable, and I’m also calling this GetCandidatePositionStart, so I’m creating a continuation object. I actually make it 60 seconds, let’s change that. I can wait for 60 seconds, 20 seconds, I have a limit of how high I can go, but I’m setting a 60 second time out. And then, very importantly, what is the continuation method? What should the continuation server call when it gets the response back from the web service, because this process is finished, so what should this thing call when it’s done? Then I go and still do my HTTP request, I make it a GET. Here notice I’m setting the endpoint directly because callouts don’t work with continuations. So I’m going to call this endpoint directly, the positions/async endpoint, you would change this URL to wherever you push this to. Then I’m going to get back the continuation ID after I make my request, and that’s going to be saved here in the controller and I’m going to return the continuation as an object back to the caller. So whenever this web service finishes, the continuation server takes care of calling this, you’ll notice I get call Continuation.getResponse, and I’m getting the response related to the continuation ID that I kicked off, this is how I’m kind of tying these things back together. And then the rest of it looks the same, get the body, turn it to a JSON object, loop through positions, and show it. So this continuation server is very, very important and I have to tie it together by linking that ID together. Let’s go ahead and save this. Great. And then let’s go ahead to VisualForce and we’re going to create a brand new VisualForce page that’s equally awesome. And let’s click New. Go back to the file system, positionspageasync, I’m going to copy everything here and we’ll take a look at that in a moment. I’m going to copy it all down here. We’ll call this PositionsPageAsync and let’s look at the code. So I’ve got my customer controller, great. I’ve got my command button, actually I have the output text here, making sure I’m referencing, you see the rerender command, so I can tell it to rerender this object, just so I’m linking these two together and then show the results. So not much of a change, but you do want to make sure you’re doing those rerender things to link it together here. I’m going to save it. Now let’s preview it. So here, this should take 5 seconds. Everything renders. Page seems like it’s done and then hopefully everything pops up here. Sure enough, there you go. So it kicked it off, the page returned control, so I can still do stuff on the VisualForce page, but it went ahead and took 5 seconds to return it, it then returned control back to my controller, I rendered it on the page. So here we show the 5 second delay, showed how asynchronous callouts can work with Apex callouts. You can’t use the named credentials, but maybe that’s a price worth paying to have this better user experience.
What are some limits to be aware of? Well you can make 20 concurrent callouts within the organization, so you do want to be careful there in not using this too much with too many people or you could run into some limits where this would queue you up. There’s also the default seconds for a timeout, and so there can be 10 as a low, as a min and kind of a floor, 120 as the ceiling. So again, as you’re doing callouts you want to make sure you’re not calling these things in a super long-running fashion unless you’re doing async or something,otherwise you’re going to hit these timeout limits. And then you can have actually 100 callouts in a single transaction, which is pretty wild, that’s a pretty complex operation, but you can do a lot of these either chained or parallel callouts within your VisualForce pages.
In this module we did an overview of the Apex callouts and over the course we looked at the key concepts, making sure you understood what Apex callouts were all about. We talked about remote site settings and using this to secure the fact and tell Salesforce that you’re going to call these sites that aren’t in its control to make sure you’re opting into this. We saw how named credentials let you actually send credentials and a URL down to your code and let you decouple the code from the credentials in the URL. We jumped into generating Apex classes from a WSDL and then consuming them, which is a pretty powerful way to get code stubs instead of doing the raw stuff. But if you want to do the raw stuff, it’s great that you can jump down to HTTP objects, using JSON and XML serialization, and be able to work with that at a very low level so that you have ultimate control. We did see how long-running callouts work from within VisualForce, which can be very powerful in creating a nice user experience, and then we discussed briefly some of the limits. I really hope you enjoyed this course. This is fun for me, it’s always great to play with technologies like this and think about new ways to link applications, that’s really such a powerful concept in the cloud today as organizations are quick to adopt these systems, but all the sudden they struggle as they’re left with silo, so you’re now in a great position to help your own business, your customers, whoever you work with, really build nice integrated solutions that are thoughtful and help the end user take advantage of all the data and all the processing available in their systems and using Salesforce as an anchor for that to collect data and view data from other systems. Drop me a line on Twitter @rseroter if you enjoyed the course. Let me know if you saw any errors you’d like me to fix, and I appreciate you taking the time to watch this course.