Category Archives: economics

So Prisoners Don’t Follow the Dilemma

Prisoners and their dilemma:

We report insights into the behavior of prisoners in dilemma situations that so famously carry their name. We compare female inmates and students in a simultaneous and a sequential Prisoner’s Dilemma. In the simultaneous Prisoner’s Dilemma, the cooperation rate among inmates exceeds the rate of cooperating students. Relative to the simultaneous dilemma, cooperation among first-movers in the sequential Prisoner’s Dilemma increases for students, but not for inmates. Students and inmates behave identically as second movers. Hence, we find a similar and significant fraction of inmates and students to hold social preferences.

Posted in economics, game theory.

Fifty Shades of Late Capitalism

Something fun to read:

“…What Fifty Shades of Grey offers is an extreme vision of late-capitalist deliverance, the American (wet) dream on performance-enhancing drugs. Just as magazines such as Penthouse, Playboy, Chic, and Oui (speaking of aspirational names) have effectively equated the moment of erotic indulgence with the ultimate consumer release, a totem of the final elevation into amoral privilege, James’s trilogy represents the latest installment in the commodified sex genre. The money shot is just that: the moment when our heroine realizes she’s been ushered into the hallowed realm of the 1 percent, once and for all.”


http://www.thebaffler.com/past/fifty_shades_of_late_capitalism

via http://critical-theory.com/read-me-fifty-shades-of-late-capitalism/

Posted in economics, internet.

Risky Kakanomics

Gloria Origgi writes:

This is an application of the theory of kakonomics, that is, the study of the rational preferences for lower-quality or mediocre outcomes, to the apparently weird results of Italian elections. The apparent irrationality of 30% of the electorate who decided to vote for Berlusconi again is explained as a perfectly rational strategy of maintaining a system of mediocre exchanges in which politicians don’t do what they have promised to do and citizens don’t pay the taxes and everybody is satisfied by the exchange. A mediocre government makes easier for mediocre citizens to do less than what they should do without feeling any breach of trust.

She argues that if you elect a crappy politician, then there is little chance of progress, which seems like a bad thing. People do this, though, because maintaining low political standards allows people to have low civic standards: if the politicians are corrupt, there is no reason to pay taxes. Likewise, the politicians who have been elected on the basis of being bad leaders have no incentive to go after tax cheats, the people who put them in office. Hence there is often a self-serving and self-maintaining aspect to making less than optimal decisions: by mutually selecting for low expectations, then everyone cooperates in forgiving bad behavior.

This account assumes that bad behavior of some sort is to be expected. If someone all of a sudden starts doing the ‘right thing’ it will be a breach of trust and violating the social norm. There would be a disincentive to repeat such a transaction again, because it challenges the stability of the assumed low quality interaction and implied forgiveness associated with it.

I like Origgi’s account of kakonomics, but I think there is something missing. The claim that localized ‘good interactions’ could threaten the status quo of bad behavior seems excessive. Criticizing someone who makes everyone else look bad does happen, but this only goes to show that the ‘right’ way of doing things is highly successful. It is the exception that proves the rule: only the people in power — those that can afford to misbehave — really benefit from maintaining the low status quo. Hence the public in general should not be as accepting of a low status quo as a social norm, though I am sure some do for exactly the reasons she stated.

This got me thinking that maybe there was another force at work here that would support a low status quo. When changing from one regime to another, it is not a simple switch from one set of outcomes to the other. There can be transitional instability, especially when dealing with governments, politics, economics, military, etc. If the transition between regimes is highly unstable (more so if things weren’t that stable to begin with) then there would be a disincentive to change: people won’t want to lose what they have, even if it is not optimal. Therefore risk associated with change can cause hyperbolic discounting of future returns, and make people prefer the status quo.

Adding high risk with the benefits of low standards could make a formidable combination. If there is a robust black market that pervades most of the society and an almost certain civil unrest given political change (throw in a heavy-handed police force, just for good measure), this could be strong incentive to not challenge an incumbent government.

Posted in economics, game theory, mind, philosophy. Tagged with , , , .

Metta World Peace, James Harden and Furbizia

Everyone is saying that Metta World Peace (the basketball player formerly known as Ron Artest) is crazy for elbowing James Harden in the face. I can’t say that I disagree, but I think there is more to the story.

Did no one else notice that James Harden walked right into World Peace while he was celebrating? Watch the video. Harden walks directly into MWP. He doesn’t do anything that would cause a foul, but if he were going to actually try to receive an inbound pass, he would have walked away from opposing players, not at them.

Instead he gets real close to an ecstatic opponent known for outbursts. I’m sure he didn’t want to get elbowed in the head, but he did put himself in a position to get fouled, to take the charge as it were. If he had only been fouled, not elbowed, by a jumping MWP then people might be talking about how Harden had cleverly gotten another foul on one of the Laker’s best players through strategic gamesmanship alone.

When asked if he would shake Harden’s hand, Metta World Peace said he wouldn’t. Everyone condemned him for this too, but I’m with MWP this time. If MWP sees Harden as having taken advantage of his celebration as a cheap way to get him in trouble, then it is understandable that he wouldn’t want to shake the man’s hand. This doesn’t excuse the elbow, but it does explain the attitude.

Posted in economics, game theory. Tagged with , .

Trembling Hands 2: Inducing Irrationality

Given an intelligent rational opponent, one who has complete information of the decision tree in a game, it may be very difficult to implement an optimal strategy. All possible moves may be accounted for and hence a stalemate may exist from the outset.

One way to proceed is to act as if your opponent may make a mistake — her hand may tremble — allowing your optimal strategy to obtain. Previously I argued that there is more too the story than merely hoping that your opponent will err at some probabilistic rate. I gave examples where errors were induced, such that they were more likely at certain times or under certain conditions.

The cannonical example given was that of furbizia (gamesmanship or guile), which causes a football opponent to make a mistake. By acting in certain ways — eg “time-wasting, physical or verbal provocation and all related psychological games, arguably even diving” — it can cause an opponent to wear out and crack. Since furbizia, by definition, occurs outside the rules and regular strategies of the game of football (and without breaking any rules), how are we to account for it?

Just Walk Away

First let’s define the Walk Away principle. The Walk Away principle states that any subgame can be walked away from, just dropped, at any point in a game. For example, during football match a player may just walk off the field and go home. Not only are all the payoffs of the game lost when a person walks away, even further payoffs, such as loss of fans and teammate’s respect, can be lost. Still, this is an option for any player at any point of a match. Imagine that news of a family member being gravely injured reached the player mid-game. Then it might make sense to just walk off the field. Another example is that of a stock portfolio. Given a certain stock portfolio, grave penalties may be incurred if money is taken out of it too soon. However, if a disaster strikes, large sums of money are needed quickly and it makes sense to liquidate the stock portfolio during those times. In this case, the economic subgame represented by the portfolio is being walked away from.

The Walk Away principle highlights that no sub-game is played in a vacuum, that there are always global variables — emergency or other unusual situations — that could radically change a person’s decision making.

Consider this scene from the movie Out of Sight: George Clooney is a fugitive and Jennifer Lopez is a marshal pursuing him. The FBI has George Clooney trapped in a hotel and Jennifer Lopez is watching one of the exits. As George Clooney goes by Jennifer Lopez, he waves at her in a friendly way – they had already been aquainted. Instead of reporting seeing him to the other officers, she freezes and does nothing, allowing Clooney to escape. Instead of playing the subgame that would have gotten her a reward by bringing in a fugitive, she, in essence, (mentally) walked away from it. From the perspective of her job and other officers, she acted irrationally. However, she decided to play a different game. That game included extending the relationship with Clooney, which would have ended had he been arrested. From that perspective, she acted rationally.

I mention this example because it shows two games that had mutually exclusive payoffs, arresting the fugitive vs. continuing the relationship, and that even though she was playing the former game, Clooney’s action — waving at her — induced her to play the latter game. Also important is that if she had not already had some relationship with Clooney, his actions would not have had such an effect on her.

Trembling hands, in these cases, may then be seen not so much as an opponent acting irrationally, but as having been induced to play a subgame with mutually exclusive payoffs from the current game (like boxing during football). However, as mentioned, it is a necessary condition that there is something prior which allows a person to be induced to play a different game. How to know who is susceptible to being so induced, and what will induce them?

Counter Intelligence and Psychological Hacking

Knowing who is susceptible and how to induce irrationality them is a counterintelligence problem. If you knew ahead of time that a company had backdated stock options, you might suspect that they would be very worried about anyone looking into their procedures. If you were then able to find a way to publicize these nefarious dealings, you could blackmail them into acting in what would otherwise be considered irrational. But the opportunity only comes with the right intelligence.

In general, unless we are personally very wealthy or in charge of a wealthy organization, we will not have an intelligence service. But this doesn’t mean we can’t gather intelligence, even over the course of a sports game. Trash talking can be seen as a psychological search strategy: by liberally insulting every competitor and saying how great you are, anyone who rises, who argues –who doesn’t walk away– may be potentially unfocussed on the game at hand. Trash talking costs very little, just some breath, but getting angry and distracted can be very costly. The more clever the trash talk, the strategic fouls, and gamesmanship in general, the wider and more sophisticated the search, the psychological hacking, becomes.

Given the results of the search, eg player 15 is most affected by being called $%!#, the gamesmanship can then be focused on the candidates most likely to react badly and at the worst possible times. This is an exploitation of the Walk Away principle. Instead of walking away from the insults, the opponent walks away from the game, if only for a few seconds: the candidate is induced into playing a subgame with mutually exclusive payoffs.

Often stakes are very high in competition and hence gathering intelligence is important. Yes the intelligence gathering and implementation/ furbizia requires effort, but as Sun Tzu notes, “Hostile armies may face each other for years, striving for the victory which is decided in a single day. This being so, to remain in ignorance of the enemy’s condition simply because one grudges the outlay of a hundred ounces of silver in honors and emoluments, is the height of inhumanity.” (The Art of War  XIII.2) What is some trash talk in the face of winning a championship, or paying off a spy when thousands of lives are on the line? Intelligence gathering and strategic implementation generally dominates simply playing by the rules.

This interpretation of furbizia/ gamesmanship as an intelligence search explains its utility and provides a way to understand furbizia in a wider setting. The high return and low relative cost explains its prevalence.

Wave-Decision Duality

Different situations and search procedures need to be combined with traditional trembling hands dynamics to better evaluate decision strategies. For example, if an intelligence search yields results according to some function of time and factors x, then the chances of an opponent making a mistake will vary according to (f(x, t), σ), where σ is the chance of a non-induced trembling hands error. If f(x, t) finds results quickly compared to the game length, then the chance of the opponent making a mistake becomes high, and a radically different strategy should be employed to maximize upon these errors. If f(x, t) predicts when a mistakes will occur, then too a strategy tailored to these circumstances will be needed.

How are we to understand the function f(x, t)? Since it represents a kind of search, we can expect that at least some instances will fall under existing research: search theory, human & machine learning, and associated disciplines have been the subject of much interest and study. However, though we do not know which search will be used to gather the intelligence, the idea that we are doing a search lets us assume that it will return a result at some varying probability.

Combining this probability with a game theory strategy yields a both decision-like and statistical (or wave) -like property for every move in a game. Considering a game tree, each node has value for irrational behavior given by the search function, and a rational decision determined by a Nash (or other) equilibrium solution.

Consequences

Backwards Induction Paradox

Equilibrium solutions to a game can be calculated with equal justification starting from the top of the game tree and working towards the payoffs or from the bottom of the game tree and working backwards from the payoffs to the initial choices. This leads to the paradox of backwards induction, which occurs when the strategy that is suggested by backwards induction is based upon parts of the game that could never be reached by using that strategy, making that strategy based upon irrational results. The traditional trembling hands solution to this paradox states that one simply assumes the rational opponent could make a mistake, have their hand tremble, justifying the reliance on the otherwise unreachable parts of the decision tree.

Now, given an associated search function in the game tree, hoping for irrationality is replaced by a search that induces it. Since this search induces irrationality, it makes sense to allow for, if not expect, irrationality. This justifies using backwards induction as a strategy even if it relies upon parts of the game that should not ever be rationally reached, especially for iterated games.

Reputation

Since the search finds information about the opponent to be used during the game — but doesn’t have to be found during the game — braggadocio and cultivating a personality off the field is a can reveal information about an opponent prior to the game, saving valuable time. Having a reputation may help prime opponents* to being induced before ever personally interacting with them. Though it may also warn opponents to be prepared, everyone should already know to be prepared due to furbizia’s high prevalence anyway, and hence little is lost by practicing gamesmanship. If anything, not playing with some degree of gamesmanship is the exception to the rule, witnessed by the comment that someone ‘plays the game the right way’ is said as a compliment: that person is so good that it is completely unnecessary to use gamesmanship.

You may not want to have a reputation for playing dirty in general, but this is outside the scope of the competitive struggle, especially if you are successful. And some people like anti-heroes and ‘bad-boys,’ so a less than straight-edge reputation is not strictly detrimental. Perhaps this interpretation of furbizia also helps explain the appeal of such types. Since furbizia is an effective strategy, there is value in being ‘bad.’ An ‘all options on the table’ strategy and reputation may also be worthwhile, too. Theoretically, then, although certain reputations can have ill effects, these consequences are not significant enough to rule out using effective strategies that lead to such reputations.

Moreover, having personality and reputation is not limited to the actual players, but may include the fans. Fans insulting or harassing opponent players, and cheering/ chanting loudly at key moments may help break an opponent’s concentration. All of these practices can help lead to a critical mistake by an opponent.

Defense and Counterintelligence

Is there a way to defend against furbizia? Though there are many types of games that allow for gamesmanship, I’ll make a few suggestions on defending against it.

One of the major differences between football/soccer and other sport is that the clock never stops in soccer. One half-time break and that is all. This is very different from, say, basketball, where time outs may be called. The time out allows players to take a break, catch their breath and reorganize.

Reorganization can break up a counter intelligence search: when the situation changes, prior searches may no longer be applicable, forcing the search to begin anew. For instance, a basketball the coach may recognize the symptoms of psychological stress and call a time out before any mistakes are made. Coaches and teammates who are able to diagnose warning signs, who understand furbizia themselves, will be able to take action such as calling time outs or separating teammates from dangerous situations before a problem occurs. Without any (or much fewer) breaks, such as in soccer, a chance to take a break and reorganize is harder to come by. Perhaps this is why furbizia in soccer is so well developed: it has the greatest opportunity to work in a game with fewer breaks.

Any and all other methods for disrupting the search, if only cracking a joke that makes the opponent laugh, calling the referee’s attention to the gamesmanship or being so friendly that your opponent feels bad insulting you, should be employed as applicable.

Alternatively, one can run counter-intelligence. If an opponent is committed to using furbizia, then the trick is to get them to over-commit to it. Either push them too far, so their gamesmanship does break an official rule of the game, or make them waste time and effort on it to the detriment of the rest of their play.  In the end, make furbizia work for you.

Posted in economics, game theory.

Trembling Hands

At least since Selten (1975) game theorists have considered that given a series of decisions there is some small probability that the person making the decisions will make a mistake and do something irrational, even if she knows the right thing to do.  This is called the trembling hand approach: although a person rationally knows the right (rational) thing to do, sometimes her hand trembles and she chooses incorrectly.

Therefore, given a game defined by a finite set of iterated decisions and payoffs in which all the rational moves are known by both players (think Tic Tac Toe), there is a ‘perturbed’ game in which the rational choices are not made.  So consider playing a game of Tic Tac Toe:  Either player can always force a draw in Tic Tac Toe and hence prevent loss.  However, it is easy enough to make a mistake (through inattentiveness, eg) and allow your opponent to win.

Tic Tac Toe Game Tree

Tic Tac Toe Game Tree showing possible decisions for the first two moves

I believe this approach is a good start but does not go nearly far enough to incorporate probability into game theory.  The issue stems from the trembling hand approach assuming that irrational behavior occurs because of ‘some unspecified psychological mechanism.’  This is fine, but then every trembling hand probability, every chance of making an irrational decision, is defined as a separate, independent probability.  This means that making an irrational decision is based on chance, as if we roll a die every decision we make.

Perhaps some people have this problem, that they act irrationally at probabilistic rates, but this doesn’t seem either realistic, or fit with the idea that a psychological mechanism was at work.  If some psychological mechanism was at work, then we would expect

  1. The probabilities of making mistakes would not be independent of each other, since they have a common source.
  2. There would be a much higher chance of irrationality at times when the psychological issue manifests itself.

One example of what I have in mind is the effectiveness of gamesmanship in sport.  Gamesmanship is the art of getting into your opponents head and causing them to make mistakes.  Consider this description of “furbizia” in Italian soccer by Andrea Tallarita:

Perhaps nothing has been more influential in determining the popular perception of the Italian game than furbizia, the art of guile… The word ‘furbizia’ itself means guile, cunning or astuteness. It refers to a method which is often (and admittedly) rather sly, a not particularly by-the-book approach to the performative, tactical and psychological part of the game. Core to furbizia is that it is executed by means of stratagems which are available to all players on the pitch, not only to one team. What are these stratagems? Here are a few: tactical fouls, taking free kicks before the goalkeeper has finished positioning himself, time-wasting, physical or verbal provocation and all related psychological games, arguably even diving… Anyone can provoke an adversary, but it takes real guile (real furbizia) to find the weakest links in the other team’s psychology, then wear them out and bite them until something or someone gives in – all without ever breaking a single rule in the book of football. (via)

If we try to explain the an instance of someone making an irrational play in a game due to gamesmanship/furbizia according to the trembling hand model, we run into difficulty.  The decision tree according to the ‘trembling hand’ theory would have a series of decisions each with a low probability of making an irrational mistake:

.01 — .01 — .01 — .01 — .01

Hence it cannot explain why someone would crack later in the game as opposed to earlier, since all the probabilities are equal.  Nor can it explain why people make irrational decisions at higher rates when playing against a crafty opponent than they would make otherwise. Therefore the trembling hand model cannot explain the effectiveness of gamesmanship.

But the decision tree given linked, non-independent probabilities might have the chance of an irrational decision given by:

.01 — .05 — .1 — .17 — .25

This model has an increasing chance of irrational action.  As time progresses, it becomes increasingly likely that an irrational choice will occur due to the gamesmanship of the opponent.

I’ll refer to this model generally as induced irrationality.  Induced irrationality occurs when the chance of making a rational decision decreases due to some factor, or when the chances of making irrational decisions over time change in concert, or both.

Other phenomena follow this pattern.  Bullying comes to mind: it is similar to gamesmanship in its breaking or bending of ‘rules’ over time to get in someone’s head and thence get them to do things they would rather not do.  The bullied will act irrationally in the presence of the bully and potentially more so as the bullying continues, perhaps even leading to “snapping”— doing something seriously irrational.

Phobias are also similar: for whatever reason a person has a phobia, and given the presence of that object or situation, the otherwise rational person will make different decisions.

Moreover this may have something to do with the Gambler’s Fallacy:   By making a gambler associate a pattern to some random act, such as by showing the gambler all the recent values of a roulette wheel in order to convince the gambler to believe that the wheel likely will land on red (or losing a few bets to a shill in 3 card monte, or seeing a pattern in the stock market, etc.), the casino has planted a belief in the gambler.  Hence, as time goes on and red is not landed upon, the gambler increasingly thinks it is ever more likely that red will hit (even though it has the same low chance as it always did). Hence the gambler will likely bet more later — more irrationally —  as he expects red to be increasingly likely to hit.

Hence, though trembling hands may be a factor in irrational decision making, it does not seem like it is the only possibility or even the most significant in a number of interesting cases.

—————————————

Selten, R. (1975). ‘Re-examination of the Perfectness Concept for Equilibrium Points in Extensive Games.’ International Journal of Game Theory, 4: 22–55.

My brother beat the Tic Tac Toe playing chicken when the Chinatown Fair Arcade (NYC) still operated.  I assume that there was a computer choosing the game moves and it happened to glitch when my brother was playing: though the machine claimed it won, if you looked at the Xs and Os, my brother had won.  We asked the manager for our promised bag of fortune cookies.  He said he didn’t actually have a bag since the chicken wasn’t ever supposed to lose.

Posted in economics, game theory. Tagged with , .

Occam’s Razor and Entropy

I was trying to understand Occam’s Razor, specifically I wanted to know its justification.  There are posts over at Wikipedia and the Stanford Encyclopedia of Philosophy worth looking at, but neither left me satisfied.

Instead, I came up with “Death Implies Economy”.  What this means is that we are fundamentally limited in time and resources, and hence we cannot afford to waste what little we have on unnecessary complication.  DIE is a metaphysical justification of ontological parsimony:  regardless of how we come to the knowledge of death, the principle only requires that we are fundamentally limited and is agnostic as to how we come to understand this of ourselves.  [One may revise the principle to ‘Demise Implies Economy’ without problem or changing acronym.]

Now, the reason I wanted to figure out Occam’s Razor was because I thought it might help me understand entropy better.  Entropy seems to be this force or cause that basically is always at work and does whatever we don’t want it to.  Jerk.  Of course the universe has no reason to conform to our way of doing things, or worse, my way of viewing the world, but entropy just seems to be excessive:  why should our physical science be subject to a form of energy loss?  This makes me think it is our fault.  No, not ‘fault’, but intrinsic part of how we go about our science.  My apologies to the universe for calling it a jerk.

So, back to Occam’s Razor and DIE.  If DIE underpins Occam’s Razor, then we are metaphysically bound to proceed in a piecemeal manner.  Even our most radical theories are not developed by immortals with no care for time.  So, in some sense, our theories are also fundamentally limited and hence will always admit some unknown factors as a metaphysical consequence.

It is fair to ask if this is all just a fancy way of stating pessimistic induction, “Since we haven’t gotten theories perfect in the past, we shouldn’t expect to in the future”?  How can I make the claim that we will never succeed in this scientific endeavor?

My answer is that these questions raise legitimate issues, but the specific question at hand is not to speculate on what will happen with future theory but how we are to understand entropy and simplicity now.  And to question whether our adherence to ontological parsimony has the theoretical consequence of an unresolvable force.   Since we must believe the theories we have, at least to some extent, whatever these theories do not describe must be left in an accordingly deep mystery– as the result of an unexplained force at least as powerful as the forces we do explain. Therefore I have to conclude that, given a metaphysical understanding of Occam’s Razor such as DIE, there is a legitimate concern of inevitable unresolvable causal consequences which could manifest as various forms of entropy.

Posted in economics, ontology, philosophy, physics, science.

Economics

Readers of this blog may have noticed a lack of updates recently. I can’t apologize: I’ve been eating, breathing and drinking philosophy for so long, that now that I have written everything I wanted to write, I feel free.  I wish it on all of you. [Happy New Year Everyone!]

But this doesn’t stop me from thinking.  I was at a Christmas party  and got talking with an Indonesian economics grad student.  He was researching economic methods Indonesia could use to become treated as a major world power, such as investing in ports with international business significance.  Interesting stuff.

Unfortunately my knowledge of economics is woeful.  However, when I pressed him to explain exactly how his economics works, he used ethical terms.  This gave me the idea that economics is fundamentally based in ethics.  When economists speak of value, this use of value is not different than the value we use in ethics, only a bit more abstracted.

Money used to represent a commitment of the issuing institution to having a certain amount of a precious metal on hand.  The cash was a proxy for that metal.  Metal, of course, has no inherent value: it is just a lump of metal.  What gives a lump of metal value is its properties that people use for specific purposes, and these purposes are fulfilling the commitments we have in daily life.  Hence money is an eventual proxy for commitments.

Now, commitments are a relativistic metaphysical substance, something I have much more experience with (man that’s a funny thing to say).  Relativistic metaphysical substances can be analyzed along the general guidelines of physical relativity: there is a general theory, there is a special theory, and then there is quantum.

The forces of macroeconomics can be conceptually aligned with a general relativistic theory, and microeconomics with special relativity.  Unfortunately everyone already understands these things so there is little hope of finding some inefficiency to exploit.

What people don’t get is quantum mechanics: it is just accepted that things are weird in the quantum world.  My view is that quantum phenomena are a highly sophisticated relativistic measurement issue (yes, I have seen all the data against this view and I am still convinced).  This allows me to look in the world of economics for similar relations and, lo and behold, impulse buying fits the schema.

Impulse buying appears to unbelievably unstudied: it has 4 whole paragraphs dedicated to it in Wikipedia.  I know this isn’t the best judge of research, but other economic topics in Wikipedia seem to have textbooks written about them and impulse buying has 1 academic journal reference (about how distraction affects brand selection during an impulse buy, not exactly the underlying theory) and a reference to a ‘Natural Parenting’ magazine article.  I feel like I can declare myself an expert right now: I am the foremost leader in the economic theory of impulse buying.

I guess now that I am done with philosophy I can start a business to see if my theories are correct…. never thought I’d end up in experimental philosophy, but it just goes to show that you can never say never.

Posted in economics, philosophy, Relativity, science. Tagged with , , , .