CloudFormation Template - any way to get a Spot-Fleet-Request ID? - json

I'm attempting to create a single template that creates the following:
AWS::EC2::SpotFleet resource
2 AWS::ApplicationAutoScaling::ScalingPolicy resources (scale up, scale down)
Initially, my template included only the SpotFleet resource, and I confirmed that the stack would create without issue. Once I added the ScalingPolicy resources, the stack would rollback because there was "No scalable target registered for namespace..." So, I added an additional resource.
AWS::ApplicationAutoScaling::ScalableTarget resource.
(From http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-applicationautoscaling-scalabletarget.html#cfn-applicationautoscaling-scalabletarget-resourceid)
{
"Type" : "AWS::ApplicationAutoScaling::ScalableTarget",
"Properties" : {
"MaxCapacity" : Integer,
"MinCapacity" : Integer,
"ResourceId" : String,
"RoleARN" : String,
"ScalableDimension" : String,
"ServiceNamespace" : String
}
}
The ResourceID is a required property. I have the data for all the other properties, but when researching what data is needed for the ResourceID property, I have found that the data I need is the spot-fleet-request ID, (something like this: "SpotFleetRequestId": "sfr-73fbd2ce-aa30-494c-8788-1cee4EXAMPLE").
So here's the problem: Since I am creating the spot fleet request in the same template as the scaling policy, I can't put the SpotFleetRequestId in manually, since to my knowledge this is created when the resource is and there's no way to anticipate what the request ID will be. In other templates, with other kinds of resources, I've simply used "Ref" or "Fn::GetAtt" to pass in the arn of a resource without having to manually input this. However--there seems to be no way to do this with a SpotFleetRequestID. All the research I've done has turned up nothing, not even a single template example that uses a method like I'm describing - the only examples available assume that the scalable target resource already exists and the SpotFleetRequestID is known prior to creating the ScalingPolicy.
Does anyone have any idea if referring to the SpotFleetRequestID of an AWS::EC2::SpotFleet initialized in the same template is even possible? Or am I just missing something REALLY obvious?
-KungFuBilly

Turns out that if you "Ref" the logical name of the AWS::EC2::SpotFleet it will return the request ID. Then, it's a matter of using "Fn::Join" to get the right data for the ResourceID. Should look something like this:
"ResourceId": {
"Fn::Join": [
"/",
[
"spot-fleet-request",
{
"Ref": "SpotFleet"
}
]
]
},
That will output: spot-fleet-request/"SpotFleetRequestID"

Related

what is view in couchbase

I am trying to understand what exactly couchbase view is used for, I have gone through some materials in docs, but the 'view' concept does not settle me quite well.
Are views in couchbase analogues to views in view in RDBMS?
https://docs.couchbase.com/server/6.0/learn/views/views-basics.html
A view performs the following on the Couchbase unstructured (or
semi-structured) data:
Extract specific fields and information from the data files.
Produce a view index of the selected information.
how does view and index work here, seems there is separate index for view. so if a documents updates are both indexes updated?
https://docs.couchbase.com/server/6.0/learn/views/views-store-data.html
In addition, the indexing of data is also affected by the view system
and the settings used when the view is accessed.
Helpful post:
Views in Couchbase
You can think of Couchbase Map/Reduce views as similar to materialized views, yes. Except that you create them with JavaScript functions (a map function and optionally a reduce function).
For example:
function(doc, meta)
{
emit(doc.name, [doc.city]);
}
This will look at every document, and save a view of each document that contains just city, and has a key of name.
For instance, let's suppose you have two documents;
[
key 1 {
"name" : "matt",
"city" : "new york",
"salary" : "100",
"bio" : "lorem ipsum dolor ... "
},
key 2 {
"name" : "emma",
"city" : "columbus",
"salary" : "120",
"bio" : "foo bar baz ... "
}
]
Then, when you 'query' this view, instead of full documents, you'll get:
[
key "matt" {
"city" : "new york"
},
key "emma" {
"city" : "columbus"
}
]
This is a very simple map. You can also use reduce functions like _count, _sum, _stats, or your own custom.
The results of this view are stored alongside the data on each node (and updated whenever the data is updated). However, you should probably stay away from Couchbase views because:
Views are stored alongside the data on each node. So when reading it, data has to be pulled from every node, combined, and pulled again. "Scatter/gather"
JavaScript map/reduce doesn't give you all the query capabilities you might want. You can't do stuff like 'joins', for instance.
Couchbase has SQL++ (aka N1QL), which is more concise, declarative, and uses global indexes (instead of scatter/gather), so it will likely be faster and reduce strains during rebalance, etc.
Are deprecated as of Couchbase Server 7.0 (and not available in Couchbase Capella at all)

ORDA: How do I specify field order when using entitySelection.toCollection()?

Background
I'm converting old 4D Database code to use new ORDA concepts introduced in v17. However, I've noticed an oddity. When I have an entitySelection that I created using ds[$vtTableName].query(), and I convert that entitySelection to a collection (using .toCollection(), the order of the fields that I specify isn't honored.
Example Code:
C_OBJECT($voSelection)
$voSelection:=ds.Users.query("Active = 'True'")
C_COLLECTION($vcUsers)
$vcUsers:=$voSelection.toCollection("FirstName, LastName, DTLastSignin")
Expected Output
I would expect $vcUsers to be a collection of objects, and that each object would look like:
{ "FirstName" : "John", "LastName" : "Smith", "DTLastSignin" : "2019-10-12T32:23:00" }
Actual Output
Instead, I'm getting a different order:
{ "DTLastSignin" : "2019-10-12T32:23:00", "FirstName" : "John", "LastName" : "Smith" }
This has broken some of my API consumers because they expect to be able to specify field order, which the old way (Selection to JSON) respects. However, toCollection() doesn't appear to.
I can't find any sort of documentation about field order and if it is even suppose to. The official documentation shows the fields respecting the order, but maybe it's just a coincidence.
The answer to this is that you can't, because the toCollection() field list is just a filter. Under the hood, the Entity's properties are being looped through in their natural order, and being filtered whether or not the field is one of the specified filter fields.
That is the reason that .toCollection is faster than the clasic Selection to JSON method.
One way around this potentially would be to use use a .map() function. However, that would degrade performance, probably. I haven't done any profiling, so I'm not sure.

How can I pull an IPv4 address from a JSON response by seacrhing nested values?

I am new to Ruby and have been stuck at this for a while now. I am getting a JSON responses as mentioned below and aim to search for the substring where the value of its substring is something as specified by me.
For example, I am getting the response below:
{
"00:00:00:CC:00:CC": {
"family": "lladdr"
},
"10.0.0.20": {
"family": "inet",
"prefixlen": "24",
"netmask": "255.255.255.0",
"broadcast": "10.0.0.255",
"scope": "Global"
},
"ff00::f00:00ff:fff0:00f0": {
"family": "inet6",
"prefixlen": "64",
"scope": "Link",
"tags": []
}
}
I need to get the value of the parent where the key family has a value equal to inet. In this case, I just want 10.0.0.20 as output when family equals inet.
I went through multiple questions here, and Google did not help. I understand that I will need to parse the JSON using JSON.parse, and then use maybe find or select to get my answer, but I was not able to get it working.
I am not sure if there is any other way I can do this like you would do in Bash using grep or awk. One hack might be to use something like foo.[46..54] which will output the IP, but again I believe that would be a bad way of solving this.
Use Hash#invert
Assuming that your Hash is already stored in response using JSON#parse, one way to solve the problem is to invert the Hash with the Hash#invert method. For example:
# Return an Array of IPv4, then pop the last/only String value.
response.invert.select { |h| h.values.include? 'inet' }.values.pop
#=> "10.0.0.20"
This is quick and simple, and it works with your provided data. However, there are some minor caveats.
Caveats
Assumes there is only one IPv4 address in the response Hash. If you have more than one key with inet as a value, don't use pop and deal with the resulting Array as you see fit. For example:
response.invert.select { |h| h.values.include? 'inet' }.values
#=> ["10.0.0.20"]
It assumes the key for each top-level JSON object is an IP address. We're not really validating anything.
It works for the JSON you have, but isn't solving for arbitrarily nested or varied data structures. If you have different kinds of inputs, consider it "some assembly required."
If you have no inet family, then {}.values.pop may return nil. Make sure you plan for that in your application.
None of these are show-stoppers for your particular use case, but they are certainly worth keeping in mind. Your mileage may vary.
If you want the family inet,
result = JSON.parse(response)
family = result.detect { |k,v| v['family'] == 'inet' }
family[0] # 10.0.0.20
Note that detect will return an array.

How do I ensure Firebase database structure using anonymous auth?

I have a public-input type app using Firebase, with anonymous auth. The user data is used to create points on a map. Each anonymous user can only edit the data inside the node matching their auth id - via security rules.
However, my app depends on a certain database structure. How do I ensure my database structure/integrity using anonymous auth, since the database url is client-side readable?
I think it is possible with security and validation rules, but I'm not sure. Maybe deny children creation in a node? This would be necessary to ensure the schema is followed.
Each auth node can have many key nodes, but I would want to limit this Firebase-side. And each key node must follow the schema below (so I can pull out the geojson easily). Below is my current setup - wondering what is missing?
"features" : {
"5AGxfaK2q8hjJsmsO3PUxUs09Sz1" : {
"-KS3R4sWPdcDkrxyIFX6" : {
"geometry" : {
"coordinates" : [ -81.88247680664062, 38.884619201291905 ],
"type" : "Point"
},
"properties" : {
"color" : "#2be",
"title" : ""
},
"type" : "Feature"
},
Authentication and database schema are completely separate topics. You ensure database schema by using a combination of .write and .validate rules in your security doc, not by anything to do with your authentication provider (i.e. Anonymous authentication).
This is described in detail in our database security guide.
A quick summary:
hasChildren: specify required fields
newData: refer to the data being written
data: refer to data already in the database
.validate: specify data schema using things like newData.isString() or newData.val() == data.val() + 1
Keep in mind that .validate rules are only run for non-null values. Thus, if you want to try something like !data.exists() (i.e. you can only write to this path once and can't modify it later) or newData.exists() (i.e. you can't delete this data) then you need to specify those in a .write rule.
Refer to the guide for more detail.

Use of PUT vs PATCH methods in REST API real life scenarios

First of all, some definitions:
PUT is defined in Section 9.6 RFC 2616:
The PUT method requests that the enclosed entity be stored under the supplied Request-URI. If the Request-URI refers to an already existing resource, the enclosed entity SHOULD be considered as a modified version of the one residing on the origin server. If the Request-URI does not point to an existing resource, and that URI is capable of being defined as a new resource by the requesting user agent, the origin server can create the resource with that URI.
PATCH is defined in RFC 5789:
The PATCH method requests that a set of changes described in the
request entity be applied to the resource identified by the Request-
URI.
Also according to RFC 2616 Section 9.1.2 PUT is Idempotent while PATCH is not.
Now let us take a look at a real example. When I do POST to /users with the data {username: 'skwee357', email: 'skwee357#domain.example'} and the server is capable of creating a resource, it will respond with 201 and resource location (lets assume /users/1) and any next call to GET /users/1 will return {id: 1, username: 'skwee357', email: 'skwee357#domain.example'}.
Now let us say I want to modify my email. Email modification is considered "a set of changes" and therefore I should PATCH /users/1 with "patch document". In my case it would be the JSON document: {email: 'skwee357#newdomain.example'}. The server then returns 200 (assuming permission are ok). This brings me to first question:
PATCH is NOT idempotent. It said so in RFC 2616 and RFC 5789. However if I issue the same PATCH request (with my new email), I will get the same resource state (with my email being modified to the requested value). Why is PATCH not then idempotent?
PATCH is a relatively new verb (RFC introduced in March 2010), and it comes to solve the problem of "patching" or modifying a set of fields. Before PATCH was introduced, everybody used PUT to update resources. But after PATCH was introduced, it leaves me confused about what PUT is used for. And this brings me to my second (and the main) question:
What is the real difference between PUT and PATCH? I have read somewhere that PUT might be used to replace entire entity under specific resource, so one should send the full entity (instead of set of attributes as with PATCH). What is the real practical usage for such case? When would you like to replace / overwrite an entity at a specific resource URI and why is such an operation not considered updating / patching the entity? The only practical use case I see for PUT is issuing a PUT on a collection, i.e. /users to replace the entire collection. Issuing PUT on a specific entity makes no sense after PATCH was introduced. Am I wrong?
NOTE: When I first spent time reading about REST, idempotence was a confusing concept to try to get right. I still didn't get it quite right in my original answer, as further comments (and Jason Hoetger's answer) have shown. For a while, I have resisted updating this answer extensively, to avoid effectively plagiarizing Jason, but I'm editing it now because, well, I was asked to (in the comments).
After reading my answer, I suggest you also read Jason Hoetger's excellent answer to this question, and I will try to make my answer better without simply stealing from Jason.
Why is PUT idempotent?
As you noted in your RFC 2616 citation, PUT is considered idempotent. When you PUT a resource, these two assumptions are in play:
You are referring to an entity, not to a collection.
The entity you are supplying is complete (the entire entity).
Let's look at one of your examples.
{ "username": "skwee357", "email": "skwee357#domain.example" }
If you POST this document to /users, as you suggest, then you might get back an entity such as
## /users/1
{
"username": "skwee357",
"email": "skwee357#domain.example"
}
If you want to modify this entity later, you choose between PUT and PATCH. A PUT might look like this:
PUT /users/1
{
"username": "skwee357",
"email": "skwee357#gmail.com" // new email address
}
You can accomplish the same using PATCH. That might look like this:
PATCH /users/1
{
"email": "skwee357#gmail.com" // new email address
}
You'll notice a difference right away between these two. The PUT included all of the parameters on this user, but PATCH only included the one that was being modified (email).
When using PUT, it is assumed that you are sending the complete entity, and that complete entity replaces any existing entity at that URI. In the above example, the PUT and PATCH accomplish the same goal: they both change this user's email address. But PUT handles it by replacing the entire entity, while PATCH only updates the fields that were supplied, leaving the others alone.
Since PUT requests include the entire entity, if you issue the same request repeatedly, it should always have the same outcome (the data you sent is now the entire data of the entity). Therefore PUT is idempotent.
Using PUT wrong
What happens if you use the above PATCH data in a PUT request?
GET /users/1
{
"username": "skwee357",
"email": "skwee357#domain.example"
}
PUT /users/1
{
"email": "skwee357#gmail.com" // new email address
}
GET /users/1
{
"email": "skwee357#gmail.com" // new email address... and nothing else!
}
(I'm assuming for the purposes of this question that the server doesn't have any specific required fields, and would allow this to happen... that may not be the case in reality.)
Since we used PUT, but only supplied email, now that's the only thing in this entity. This has resulted in data loss.
This example is here for illustrative purposes -- don't ever actually do this (unless your intent is to drop the omitted fields, of course... then you are using PUT as it should be used). This PUT request is technically idempotent, but that doesn't mean it isn't a terrible, broken idea.
How can PATCH be idempotent?
In the above example, PATCH was idempotent. You made a change, but if you made the same change again and again, it would always give back the same result: you changed the email address to the new value.
GET /users/1
{
"username": "skwee357",
"email": "skwee357#domain.example"
}
PATCH /users/1
{
"email": "skwee357#gmail.com" // new email address
}
GET /users/1
{
"username": "skwee357",
"email": "skwee357#gmail.com" // email address was changed
}
PATCH /users/1
{
"email": "skwee357#gmail.com" // new email address... again
}
GET /users/1
{
"username": "skwee357",
"email": "skwee357#gmail.com" // nothing changed since last GET
}
My original example, fixed for accuracy
I originally had examples that I thought were showing non-idempotency, but they were misleading / incorrect. I am going to keep the examples, but use them to illustrate a different thing: that multiple PATCH documents against the same entity, modifying different attributes, do not make the PATCHes non-idempotent.
Let's say that at some past time, a user was added. This is the state that you are starting from.
{
"id": 1,
"name": "Sam Kwee",
"email": "skwee357#olddomain.example",
"address": "123 Mockingbird Lane",
"city": "New York",
"state": "NY",
"zip": "10001"
}
After a PATCH, you have a modified entity:
PATCH /users/1
{"email": "skwee357#newdomain.example"}
{
"id": 1,
"name": "Sam Kwee",
"email": "skwee357#newdomain.example", // the email changed, yay!
"address": "123 Mockingbird Lane",
"city": "New York",
"state": "NY",
"zip": "10001"
}
If you then repeatedly apply your PATCH, you will continue to get the same result: the email was changed to the new value. A goes in, A comes out, therefore this is idempotent.
An hour later, after you have gone to make some coffee and take a break, someone else comes along with their own PATCH. It seems the Post Office has been making some changes.
PATCH /users/1
{"zip": "12345"}
{
"id": 1,
"name": "Sam Kwee",
"email": "skwee357#newdomain.example", // still the new email you set
"address": "123 Mockingbird Lane",
"city": "New York",
"state": "NY",
"zip": "12345" // and this change as well
}
Since this PATCH from the post office doesn't concern itself with email, only zip code, if it is repeatedly applied, it will also get the same result: the zip code is set to the new value. A goes in, A comes out, therefore this is also idempotent.
The next day, you decide to send your PATCH again.
PATCH /users/1
{"email": "skwee357#newdomain.example"}
{
"id": 1,
"name": "Sam Kwee",
"email": "skwee357#newdomain.example",
"address": "123 Mockingbird Lane",
"city": "New York",
"state": "NY",
"zip": "12345"
}
Your patch has the same effect it had yesterday: it set the email address. A went in, A came out, therefore this is idempotent as well.
What I got wrong in my original answer
I want to draw an important distinction (something I got wrong in my original answer). Many servers will respond to your REST requests by sending back the new entity state, with your modifications (if any). So, when you get this response back, it is different from the one you got back yesterday, because the zip code is not the one you received last time. However, your request was not concerned with the zip code, only with the email. So your PATCH document is still idempotent - the email you sent in PATCH is now the email address on the entity.
So when is PATCH not idempotent, then?
For a full treatment of this question, I again refer you to Jason Hoetger's answer which already fully answers that.
Though Dan Lowe's excellent answer very thoroughly answered the OP's question about the difference between PUT and PATCH, its answer to the question of why PATCH is not idempotent is not quite correct.
To show why PATCH isn't idempotent, it helps to start with the definition of idempotence (from Wikipedia):
The term idempotent is used more comprehensively to describe an operation that will produce the same results if executed once or multiple times [...] An idempotent function is one that has the property f(f(x)) = f(x) for any value x.
In more accessible language, an idempotent PATCH could be defined as: After PATCHing a resource with a patch document, all subsequent PATCH calls to the same resource with the same patch document will not change the resource.
Conversely, a non-idempotent operation is one where f(f(x)) != f(x), which for PATCH could be stated as: After PATCHing a resource with a patch document, subsequent PATCH calls to the same resource with the same patch document do change the resource.
To illustrate a non-idempotent PATCH, suppose there is a /users resource, and suppose that calling GET /users returns a list of users, currently:
[{ "id": 1, "username": "firstuser", "email": "firstuser#example.org" }]
Rather than PATCHing /users/{id}, as in the OP's example, suppose the server allows PATCHing /users. Let's issue this PATCH request:
PATCH /users
[{ "op": "add", "username": "newuser", "email": "newuser#example.org" }]
Our patch document instructs the server to add a new user called newuser to the list of users. After calling this the first time, GET /users would return:
[{ "id": 1, "username": "firstuser", "email": "firstuser#example.org" },
{ "id": 2, "username": "newuser", "email": "newuser#example.org" }]
Now, if we issue the exact same PATCH request as above, what happens? (For the sake of this example, let's assume that the /users resource allows duplicate usernames.) The "op" is "add", so a new user is added to the list, and a subsequent GET /users returns:
[{ "id": 1, "username": "firstuser", "email": "firstuser#example.org" },
{ "id": 2, "username": "newuser", "email": "newuser#example.org" },
{ "id": 3, "username": "newuser", "email": "newuser#example.org" }]
The /users resource has changed again, even though we issued the exact same PATCH against the exact same endpoint. If our PATCH is f(x), f(f(x)) is not the same as f(x), and therefore, this particular PATCH is not idempotent.
Although PATCH isn't guaranteed to be idempotent, there's nothing in the PATCH specification to prevent you from making all PATCH operations on your particular server idempotent. RFC 5789 even anticipates advantages from idempotent PATCH requests:
A PATCH request can be issued in such a way as to be idempotent,
which also helps prevent bad outcomes from collisions between two
PATCH requests on the same resource in a similar time frame.
In Dan's example, his PATCH operation is, in fact, idempotent. In that example, the /users/1 entity changed between our PATCH requests, but not because of our PATCH requests; it was actually the Post Office's different patch document that caused the zip code to change. The Post Office's different PATCH is a different operation; if our PATCH is f(x), the Post Office's PATCH is g(x). Idempotence states that f(f(f(x))) = f(x), but makes no guarantes about f(g(f(x))).
TLDR - Dumbed Down Version
PUT => Set all new attributes for an existing resource.
PATCH => Partially update an existing resource (not all attributes required).
I was curious about this as well and found a few interesting articles. I may not answer your question to its full extent, but this at least provides some more information.
http://restful-api-design.readthedocs.org/en/latest/methods.html
The HTTP RFC specifies that PUT must take a full new resource
representation as the request entity. This means that if for example
only certain attributes are provided, those should be remove (i.e. set
to null).
Given that, then a PUT should send the entire object. For instance,
/users/1
PUT {id: 1, username: 'skwee357', email: 'newemail#domain.example'}
This would effectively update the email. The reason PUT may not be too effective is that your only really modifying one field and including the username is kind of useless. The next example shows the difference.
/users/1
PUT {id: 1, email: 'newemail#domain.example'}
Now, if the PUT was designed according the spec, then the PUT would set the username to null and you would get the following back.
{id: 1, username: null, email: 'newemail#domain.example'}
When you use a PATCH, you only update the field you specify and leave the rest alone as in your example.
The following take on the PATCH is a little different than I have never seen before.
http://williamdurand.fr/2014/02/14/please-do-not-patch-like-an-idiot/
The difference between the PUT and PATCH requests is reflected in the
way the server processes the enclosed entity to modify the resource
identified by the Request-URI. In a PUT request, the enclosed entity
is considered to be a modified version of the resource stored on the
origin server, and the client is requesting that the stored version be
replaced. With PATCH, however, the enclosed entity contains a set of
instructions describing how a resource currently residing on the
origin server should be modified to produce a new version. The PATCH
method affects the resource identified by the Request-URI, and it also
MAY have side effects on other resources; i.e., new resources may be
created, or existing ones modified, by the application of a PATCH.
PATCH /users/123
[
{ "op": "replace", "path": "/email", "value": "new.email#example.org" }
]
You are more or less treating the PATCH as a way to update a field. So instead of sending over the partial object, you're sending over the operation. i.e. Replace email with value.
The article ends with this.
It is worth mentioning that PATCH is not really designed for truly REST
APIs, as Fielding's dissertation does not define any way to partially
modify resources. But, Roy Fielding himself said that PATCH was
something [he] created for the initial HTTP/1.1 proposal because
partial PUT is never RESTful. Sure you are not transferring a complete
representation, but REST does not require representations to be
complete anyway.
Now, I don't know if I particularly agree with the article as many commentators point out. Sending over a partial representation can easily be a description of the changes.
For me, I am mixed on using PATCH. For the most part, I will treat PUT as a PATCH since the only real difference I have noticed so far is that PUT "should" set missing values to null. It may not be the 'most correct' way to do it, but good luck coding perfect.
tl;dr version
POST: is used to create an entity
PUT: is used to update/replace an existing entity where you must send the entire representation of the entity as you wish for it to be stored
PATCH: is used to update an entity where you send only the fields that need to be updated
The difference between PUT and PATCH is that:
PUT is required to be idempotent. In order to achieve that, you have to put the entire complete resource in the request body.
PATCH can be non-idempotent. Which implies it can also be idempotent in some cases, such as the cases you described.
PATCH requires some "patch language" to tell the server how to modify the resource. The caller and the server need to define some "operations" such as "add", "replace", "delete". For example:
GET /contacts/1
{
"id": 1,
"name": "Sam Kwee",
"email": "skwee357#olddomain.example",
"state": "NY",
"zip": "10001"
}
PATCH /contacts/1
{
[{"operation": "add", "field": "address", "value": "123 main street"},
{"operation": "replace", "field": "email", "value": "abc#myemail.example"},
{"operation": "delete", "field": "zip"}]
}
GET /contacts/1
{
"id": 1,
"name": "Sam Kwee",
"email": "abc#myemail.example",
"state": "NY",
"address": "123 main street",
}
Instead of using explicit "operation" fields, the patch language can make it implicit by defining conventions like:
in the PATCH request body:
The existence of a field means "replace" or "add" that field.
If the value of a field is null, it means delete that field.
With the above convention, the PATCH in the example can take the following form:
PATCH /contacts/1
{
"address": "123 main street",
"email": "abc#myemail.example",
"zip":
}
Which looks more concise and user-friendly. But the users need to be aware of the underlying convention.
With the operations I mentioned above, the PATCH is still idempotent. But if you define operations like: "increment" or "append", you can easily see it won't be idempotent anymore.
In my humble opinion, idempotence means:
PUT:
I send a compete resource definition, so - the resulting resource state is exactly as defined by PUT params. Each and every time I update the resource with the same PUT params - the resulting state is exactly the same.
PATCH:
I sent only part of the resource definition, so it might happen other users are updating this resource's OTHER parameters in a meantime. Consequently - consecutive patches with the same parameters and their values might result with different resource state. For instance:
Presume an object defined as follows:
CAR:
- color: black,
- type: sedan,
- seats: 5
I patch it with:
{color: 'red'}
The resulting object is:
CAR:
- color: red,
- type: sedan,
- seats: 5
Then, some other users patches this car with:
{type: 'hatchback'}
so, the resulting object is:
CAR:
- color: red,
- type: hatchback,
- seats: 5
Now, if I patch this object again with:
{color: 'red'}
the resulting object is:
CAR:
- color: red,
- type: hatchback,
- seats: 5
What is DIFFERENT to what I've got previously!
This is why PATCH is not idempotent while PUT is idempotent.
I might be a bit off topic considering your questions about idempotency, but I'd like you to consider evolutivity.
Consider you have the following element :
{
"username": "skwee357",
"email": "skwee357#domain.example"
}
If you modify with PUT, you have to give the whole representation of the object :
PUT /users/1
{
"username": "skwee357",
"email": "skwee357#newdomain.example"
}
Now you update the schema, and add a field phone :
PUT /users/1
{
"username": "skwee357",
"email": "skwee357#newdomain.example",
"phone": "123-456-7890"
}
Now update it again with PUT the same way, it will set phone to null. To avoid that bad side-effect, you have to update all the components that modify elements everytime you update your schema. Lame.
By using PATCH, you do not have this problem, because PATCH only updates the given fields. So, in my opinion, you should use PATCH to modify an element (whether it is really idempotent or not). That's a real-life return of experience.
Everyone else has answered the PUT vs PATCH. I was just going to answer what part of the title of the original question asks: "... in REST API real life scenarios". In the real world, this happened to me with internet application that had a RESTful server and a relational database with a Customer table that was "wide" (about 40 columns). I mistakenly used PUT but had assumed it was like a SQL Update command and had not filled out all the columns. Problems: 1) Some columns were optional (so blank was valid answer), 2) many columns rarely changed, 3) some columns the user was not allowed to change such as time stamp of Last Purchase Date, 4) one column was a free-form text "Comments" column that users diligently filled with half-page customer services comments like spouses name to ask about OR usual order, 5) I was working on an internet app at time and there was worry about packet size.
The disadvantage of PUT is that it forces you to send a large packet of info (all columns including the entire Comments column, even though only a few things changed) AND multi-user issue of 2+ users editing the same customer simultaneously (so last one to press Update wins). The disadvantage of PATCH is that you have to keep track on the view/screen side of what changed and have some intelligence to send only the parts that changed. Patch's multi-user issue is limited to editing the same column(s) of same customer.
Let me quote and comment more closely the RFC 7231 section 4.2.2, already cited in earlier comments:
A request method is considered "idempotent" if the intended effect on
the server of multiple identical requests with that method is the same
as the effect for a single such request. Of the request methods
defined by this specification, PUT, DELETE, and safe request methods
are idempotent.
(...)
Idempotent methods are distinguished because the request can be
repeated automatically if a communication failure occurs before the
client is able to read the server's response. For example, if a
client sends a PUT request and the underlying connection is closed
before any response is received, then the client can establish a new
connection and retry the idempotent request. It knows that repeating
the request will have the same intended effect, even if the original
request succeeded, though the response might differ.
So, what should be "the same" after a repeated request of an idempotent method? Not the server state, nor the server response, but the intended effect. In particular, the method should be idempotent "from the point of view of the client". Now, I think that this point of view shows that the last example in Dan Lowe's answer, which I don't want to plagiarize here, indeed shows that a PATCH request can be non-idempotent (in a more natural way than the example in Jason Hoetger's answer).
Indeed, let's make the example slightly more precise by making explicit one possible intend for the first client. Let's say that this client goes through the list of users with the project to check their emails and zip codes. He starts with user 1, notices that the zip is right but the email is wrong. He decides to correct this with a PATCH request, which is fully legitimate, and sends only
PATCH /users/1
{"email": "skwee357#newdomain.example"}
since this is the only correction. Now, the request fails because of some network issue and is re-submitted automatically a couple of hours later. In the meanwhile, another client has (erroneously) modified the zip of user 1. Then, sending the same PATCH request a second time does not achieve the intended effect of the client, since we end up with an incorrect zip. Hence the method is not idempotent in the sense of the RFC.
If instead the client uses a PUT request to correct the email, sending to the server all properties of user 1 along with the email, his intended effect will be achieved even if the request has to be re-sent later and user 1 has been modified in the meanwhile --- since the second PUT request will overwrite all changes since the first request.
To conclude the discussion on the idempotency, I should note that one can define idempotency in the REST context in two ways. Let's first formalize a few things:
A resource is a function with its co-domain being the class of strings. In other words, a resource is a subset of String × Any, where all the keys are unique. Let's call the class of the resources Res.
A REST operation on resources, is a function f(x: Res, y: Res): Res. Two examples of REST operations are:
PUT(x: Res, y: Res): Res = x, and
PATCH(x: Res, y: Res): Res, which works like PATCH({a: 2}, {a: 1, b: 3}) == {a: 2, b: 3}.
(This definition is specifically designed to argue about PUT and POST, and e.g. doesn't make much sense on GET and POST, as it doesn't care about persistence).
Now, by fixing x: Res (informatically speaking, using currying), PUT(x: Res) and PATCH(x: Res) are univariate functions of type Res → Res.
A function g: Res → Res is called globally idempotent, when g ○ g == g, i.e. for any y: Res, g(g(y)) = g(y).
Let x: Res a resource, and k = x.keys. A function g = f(x) is called left idempotent, when for each y: Res, we have g(g(y))|ₖ == g(y)|ₖ. It basically means that the result should be same, if we look at the applied keys.
So, PATCH(x) is not globally idempotent, but is left idempotent. And left idempotency is the thing that matters here: if we patch a few keys of the resource, we want those keys to be same if we patch it again, and we don't care about the rest of the resource.
And when RFC is talking about PATCH not being idempotent, it is talking about global idempotency. Well, it's good that it's not globally idempotent, otherwise it would have been a broken operation.
Now, Jason Hoetger's answer is trying to demonstrate that PATCH is not even left idempotent, but it's breaking too many things to do so:
First of all, PATCH is used on a set, although PATCH is defined to work on maps / dictionaries / key-value objects.
If someone really wants to apply PATCH to sets, then there is a natural translation that should be used: t: Set<T> → Map<T, Boolean>, defined with x in A iff t(A)(x) == True. Using this definition, patching is left idempotent.
In the example, this translation was not used, instead, the PATCH works like a POST. First of all, why is an ID generated for the object? And when is it generated? If the object is first compared to the elements of the set, and if no matching object is found, then the ID is generated, then again the program should work differently ({id: 1, email: "me#site.example"} must match with {email: "me#site.example"}, otherwise the program is always broken and the PATCH cannot possibly patch). If the ID is generated before checking against the set, again the program is broken.
One can make examples of PUT being non-idempotent with breaking half of the things that are broken in this example:
An example with generated additional features would be versioning. One may keep record of the number of changes on a single object. In this case, PUT is not idempotent: PUT /user/12 {email: "me#site.example"} results in {email: "...", version: 1} the first time, and {email: "...", version: 2} the second time.
Messing with the IDs, one may generate a new ID every time the object is updated, resulting in a non-idempotent PUT.
All the above examples are natural examples that one may encounter.
My final point is, that PATCH should not be globally idempotent, otherwise won't give you the desired effect. You want to change the email address of your user, without touching the rest of the information, and you don't want to overwrite the changes of another party accessing the same resource.
A very nice explanation is here-
https://blog.segunolalive.com/posts/restful-api-design-%E2%80%94-put-vs-patch/#:~:text=RFC%205789,not%20required%20to%20be%20idempotent.
A Normal Payload-
// House on plot 1
{
address: 'plot 1',
owner: 'segun',
type: 'duplex',
color: 'green',
rooms: '5',
kitchens: '1',
windows: 20
}
PUT For Updated-
// PUT request payload to update windows of House on plot 1
{
address: 'plot 1',
owner: 'segun',
type: 'duplex',
color: 'green',
rooms: '5',
kitchens: '1',
windows: 21
}
Note: In above payload we are trying to update windows from 20 to 21.
Now see the PATH payload-
// Patch request payload to update windows on the House
{
windows: 21
}
Since PATCH is not idempotent, failed requests are not automatically re-attempted on the network. Also, if a PATCH request is made to a non-existent url e.g attempting to replace the front door of a non-existent building, it should simply fail without creating a new resource unlike PUT, which would create a new one using the payload. Come to think of it, it’ll be odd having a lone door at a house address.
PUT method is ideal to update data in tabular format like in a relational db or entity like storage. Based on use case it can be used to update data partially or replace the entity as a whole. This will always be idempotent.
PATCH method can be used to update(or restructure) data in json or xml format which is stored in local file system or no sql database. This can be performed by mentioning the action/operation to be performed in the request like adding/removing/moving a key-value pair to json object. The remove operation can be used to delete a key-value pair and duplicate request will result in error as the key was deleted earlier making it a non-idempotent method. refer RFC 6902 for json data patching request.
This artical has detailed information related to PATCH method.
I will try to summarize in layman terms what I understood (maybe it helps)
Patch is not fully idempotent (it can be in an ideal situation where nobody changes another field of your entity).
In an not ideal (real life) situation somebody modifies another field of your object by another Patch operation and then both operations are not Idempotent (meaning that the resource you are both modifying comes back "wrong" from either one point of view)
So you cannot call it Idempotent if it does not cover 100% of the situations.
Maybe this is not that important to some, but to others is
One additional information I just one to add is that a PATCH request use less bandwidth compared to a PUT request since just a part of the data is sent not the whole entity. So just use a PATCH request for updates of specific records like (1-3 records) while PUT request for updating a larger amount of data. That is it, don't think too much or worry about it too much.