A Method for Learning Macro-Actions for Virtual Characters Using Programming by Demonstration and Reinforcement Learning
-
- Journal of Information Processing Systems
- /
- v.8 no.3
- /
- pp.409-420
- /
- 2012
The decision-making by agents in games is commonly based on reinforcement learning. To improve the quality of agents, it is necessary to solve the problems of the time and state space that are required for learning. Such problems can be solved by Macro-Actions, which are defined and executed by a sequence of primitive actions. In this line of research, the learning time is reduced by cutting down the number of policy decisions by agents. Macro-Actions were originally defined as combinations of the same primitive actions. Based on studies that showed the generation of Macro-Actions by learning, Macro-Actions are now thought to consist of diverse kinds of primitive actions. However an enormous amount of learning time and state space are required to generate Macro-Actions. To resolve these issues, we can apply insights from studies on the learning of tasks through Programming by Demonstration (PbD) to generate Macro-Actions that reduce the learning time and state space. In this paper, we propose a method to define and execute Macro-Actions. Macro-Actions are learned from a human subject via PbD and a policy is learned by reinforcement learning. In an experiment, the proposed method was applied to a car simulation to verify the scalability of the proposed method. Data was collected from the driving control of a human subject, and then the Macro-Actions that are required for running a car were generated. Furthermore, the policy that is necessary for driving on a track was learned. The acquisition of Macro-Actions by PbD reduced the driving time by about 16% compared to the case in which Macro-Actions were directly defined by a human subject. In addition, the learning time was also reduced by a faster convergence of the optimum policies.
Increasing attention is being devoted to the mechanisms by which cells receive signals and then translate these into decisions for growth, death, or migration. Recent findings have presented significant breakthroughs in developing a deeper understanding of the activation or repression of target genes and proteins in response to various stimuli and of how they are assembled during signal transduction in cancer cells. Detailed mechanistic insights have unveiled new maps of linear and integrated signal transduction cascades, but the multifaceted nature of the pathways remains unclear. Although new layers of information are being added regarding mechanisms underlying ovarian cancer and how polymorphisms in VDR gene influence its development, the findings of this research must be sequentially collected and re-interpreted. We divide this multi-component review into different segments: how vitamin D modulates molecular network in ovarian cancer cells, how ovarian cancer is controlled by tumor suppressors and oncogenic miRNAs and finally how vitamin D signaling regulates miRNA expression. Intra/inter-population variability is insufficiently studied and a better understanding of genetics of population will be helpful in getting a step closer to personalized medicine.
The author Present a modified multiloop algorithm including feedforward for controlling a 45kW step down chopper in the power supply of Maglev. The control law for the duty cycle consists of three terms. The first is the feedforward term which compensates for variations in the input voltage. The second term consists of the difference between the slowly moving inductor current and output current. The third term consists of proportional and integral terms involving the perturbation in the output voltage. This perturvation is derived by subtracting the desired output voltage from the actual output voltage. The proportional and integral action stabilizes the system and minimizes output voltage error. To verify the validity of the proposed multiloop controller, simulation study was tried using Matlab/sirnulink.
Cullin4-RING ubiquitin ligase (CRL4) is a family of multi-subunit E3 ligases. To investigate the possible involvement of CRL4 in heat stress response, we screened T-DNA insertion mutants of putative CRL4 substrate receptors that exhibited altered patterns in response to heat stress. One of the mutants exhibited heat stress tolerance and was named heat stress tolerant DWD1 (htd1). Introduction of HTD1 gene into htd1-1 led to recovery of heat sensitivity to the wild type level, confirming that the decrease of HTD1 transcripts resulted in heat tolerance. Therefore, HTD1 plays a negative role in thermotolerance in Arabidopsis. Additionally, HTD1 directly interacted with DDB1a in yeast two-hybrid assays and associated with DDB1b in vivo, supporting that it could be a part of a CRL4 complex. Various heat-inducible genes such as HSP14.7, HSP21, At2g03020 and WRKY28 were hyper-induced in htd1-1, indicating that HTD1 could function as a negative regulator for the expression of such genes and that these genes might contribute to thermotolerance of htd1-1, at least in part. HTD1 was associated with HSP90-1, a crucial regulator of thermotolerance, in vivo, even though the decrease of HTD1 did not affect the accumulation pattern of HSP90-1 in Arabidopsis. These findings indicate that a negative role of HTD1 in thermotolerance might be achieved through its association with HSP90-1, possibly by disturbing the action of HSP90-1, not by the degradation of HSP90-1. This study will serve as an important step toward understanding of the functional connection between CRL4-mediated processes and plant heat stress signaling.
Recently, the CO2 emission and energy consumption have become critical global issues to decide the future of nations. Especially, the spread of IT products and the increased use of internet and web applications result in the energy consumption and CO2 emission of IT industry though information technologies drive global economic growth. EU, the United States, Japan and other developed countries are using IT related environmental regulations such as WEEE(Waste Electrical and Electronic Equipment), RoHS(Restriction of the use of Certain Hazardous Substance), REACH(Registration, Evaluation, Authorization and Restriction of CHemicals) and EuP(Energy using Product), and have established systematic green business/IT strategies to enhance the competitiveness of IT industry. For example, the Japan government proposed the "Green IT initiative" for being compatible with economic growth and environmental protection. Not only energy saving technologies but energy saving systems have been developed for accomplishing sustainable development. Korea's CO2 emission and energy consumption continuously have grown at comparatively high rates. They are related to its industrial structure depending on high energy-consuming industries such as iron and steel Industry, automotive industry, shipbuilding industry, semiconductor industry, and so on. In particular, export proportion of IT manufacturing is quite high in Korea. For example, the global market share of the semiconductor such as DRAM was about 80% in 2008. Accordingly, Korea needs to establish a systematic strategy to respond to the global environmental regulations and to maintain competitiveness in the IT industry. However, green competitiveness of Korea ranked 11th among 15 major countries and R&D budget for green technology is not large enough to develop energy-saving technologies for infrastructure and value chain of low-carbon society though that grows at high rates. Moreover, there are no concrete action plans in Korea. This research aims to deduce the priorities of the Korean green business/IT strategies to use multi attribute weighted average method. We selected a panel of 19 experts who work at the green business related firms such as HP, IBM, Fujitsu and so on, and selected six assessment indices such as the urgency of the technology development, the technology gap between Korea and the developed countries, the effect of import substitution, the spillover effect of technology, the market growth, and the export potential of the package or stand-alone products by existing literature review. We submitted questionnaires at approximately weekly intervals to them for priorities of the green business/IT strategies. The strategies broadly classify as follows. The first strategy which consists of the green business/IT policy and standardization, process and performance management and IT industry and legislative alignment relates to government's role in the green economy. The second strategy relates to IT to support environment sustainability such as the travel and ways of working management, printer output and recycling, intelligent building, printer rationalization and collaboration and connectivity. The last strategy relates to green IT systems, services and usage such as the data center consolidation and energy management, hardware recycle decommission, server and storage virtualization, device power management, and service supplier management. All the questionnaires were assessed via a five-point Likert scale ranging from "very little" to "very large." Our findings show that the IT to support environment sustainability is prior to the other strategies. In detail, the green business /IT policy and standardization is the most important in the government's role. The strategies of intelligent building and the travel and ways of working management are prior to the others for supporting environment sustainability. Finally, the strategies for the data center consolidation and energy management and server and storage virtualization have the huge influence for green IT systems, services and usage This research results the following implications. The amount of energy consumption and CO2 emissions of IT equipment including electrical business equipment will need to be clearly indicated in order to manage the effect of green business/IT strategy. And it is necessary to develop tools that measure the performance of green business/IT by each step. Additionally, intelligent building could grow up in energy-saving, growth of low carbon and related industries together. It is necessary to expand the affect of virtualization though adjusting and controlling the relationship between the management teams.
This article provides training exercises for executives into interpreting subroutine maps of executives' thinking in processing business and industrial marketing problems and opportunities. This study builds on premises that Schank proposes about learning and teaching including (1) learning occurs by experiencing and the best instruction offers learners opportunities to distill their knowledge and skills from interactive stories in the form of goal.based scenarios, team projects, and understanding stories from experts. Also, (2) telling does not lead to learning because learning requires action-training environments should emphasize active engagement with stories, cases, and projects. Each training case study includes executive exposure to decision system analysis (DSA). The training case requires the executive to write a "Briefing Report" of a DSA map. Instructions to the executive trainee in writing the briefing report include coverage in the briefing report of (1) details of the essence of the DSA map and (2) a statement of warnings and opportunities that the executive map reader interprets within the DSA map. The length maximum for a briefing report is 500 words-an arbitrary rule that works well in executive training programs. Following this introduction, section two of the article briefly summarizes relevant literature on how humans think within contexts in response to problems and opportunities. Section three illustrates the creation and interpreting of DSA maps using a training exercise in pricing a chemical product to different OEM (original equipment manufacturer) customers. Section four presents a training exercise in pricing decisions by a petroleum manufacturing firm. Section five presents a training exercise in marketing strategies by an office furniture distributer along with buying strategies by business customers. Each of the three training exercises is based on research into information processing and decision making of executives operating in marketing contexts. Section six concludes the article with suggestions for use of this training case and for developing additional training cases for honing executives' decision-making skills. Todd and Gigerenzer propose that humans use simple heuristics because they enable adaptive behavior by exploiting the structure of information in natural decision environments. "Simplicity is a virtue, rather than a curse". Bounded rationality theorists emphasize the centrality of Simon's proposition, "Human rational behavior is shaped by a scissors whose blades are the structure of the task environments and the computational capabilities of the actor". Gigerenzer's view is relevant to Simon's environmental blade and to the environmental structures in the three cases in this article, "The term environment, here, does not refer to a description of the total physical and biological environment, but only to that part important to an organism, given its needs and goals." The present article directs attention to research that combines reports on the structure of task environments with the use of adaptive toolbox heuristics of actors. The DSA mapping approach here concerns the match between strategy and an environment-the development and understanding of ecological rationality theory. Aspiration adaptation theory is central to this approach. Aspiration adaptation theory models decision making as a multi-goal problem without aggregation of the goals into a complete preference order over all decision alternatives. The three case studies in this article permit the learner to apply propositions in aspiration level rules in reaching a decision. Aspiration adaptation takes the form of a sequence of adjustment steps. An adjustment step shifts the current aspiration level to a neighboring point on an aspiration grid by a change in only one goal variable. An upward adjustment step is an increase and a downward adjustment step is a decrease of a goal variable. Creating and using aspiration adaptation levels is integral to bounded rationality theory. The present article increases understanding and expertise of both aspiration adaptation and bounded rationality theories by providing learner experiences and practice in using propositions in both theories. Practice in ranking CTSs and writing TOP gists from DSA maps serves to clarify and deepen Selten's view, "Clearly, aspiration adaptation must enter the picture as an integrated part of the search for a solution." The body of "direct research" by Mintzberg, Gladwin's ethnographic decision tree modeling, and Huff's work on mapping strategic thought are suggestions on where to look for research that considers both the structure of the environment and the computational capabilities of the actors making decisions in these environments. Such research on bounded rationality permits both further development of theory in how and why decisions are made in real life and the development of learning exercises in the use of heuristics occurring in natural environments. The exercises in the present article encourage learning skills and principles of using fast and frugal heuristics in contexts of their intended use. The exercises respond to Schank's wisdom, "In a deep sense, education isn't about knowledge or getting students to know what has happened. It is about getting them to feel what has happened. This is not easy to do. Education, as it is in schools today, is emotionless. This is a huge problem." The three cases and accompanying set of exercise questions adhere to Schank's view, "Processes are best taught by actually engaging in them, which can often mean, for mental processing, active discussion."
The wall shear stress in the vicinity of end-to end anastomoses under steady flow conditions was measured using a flush-mounted hot-film anemometer(FMHFA) probe. The experimental measurements were in good agreement with numerical results except in flow with low Reynolds numbers. The wall shear stress increased proximal to the anastomosis in flow from the Penrose tubing (simulating an artery) to the PTFE: graft. In flow from the PTFE graft to the Penrose tubing, low wall shear stress was observed distal to the anastomosis. Abnormal distributions of wall shear stress in the vicinity of the anastomosis, resulting from the compliance mismatch between the graft and the host artery, might be an important factor of ANFH formation and the graft failure. The present study suggests a correlation between regions of the low wall shear stress and the development of anastomotic neointimal fibrous hyperplasia(ANPH) in end-to-end anastomoses. 30523 T00401030523 ^x Air pressure decay(APD) rate and ultrafiltration rate(UFR) tests were performed on new and saline rinsed dialyzers as well as those roused in patients several times. C-DAK 4000 (Cordis Dow) and CF IS-11 (Baxter Travenol) reused dialyzers obtained from the dialysis clinic were used in the present study. The new dialyzers exhibited a relatively flat APD, whereas saline rinsed and reused dialyzers showed considerable amount of decay. C-DAH dialyzers had a larger APD(11.70
The wall shear stress in the vicinity of end-to end anastomoses under steady flow conditions was measured using a flush-mounted hot-film anemometer(FMHFA) probe. The experimental measurements were in good agreement with numerical results except in flow with low Reynolds numbers. The wall shear stress increased proximal to the anastomosis in flow from the Penrose tubing (simulating an artery) to the PTFE: graft. In flow from the PTFE graft to the Penrose tubing, low wall shear stress was observed distal to the anastomosis. Abnormal distributions of wall shear stress in the vicinity of the anastomosis, resulting from the compliance mismatch between the graft and the host artery, might be an important factor of ANFH formation and the graft failure. The present study suggests a correlation between regions of the low wall shear stress and the development of anastomotic neointimal fibrous hyperplasia(ANPH) in end-to-end anastomoses. 30523 T00401030523 ^x Air pressure decay(APD) rate and ultrafiltration rate(UFR) tests were performed on new and saline rinsed dialyzers as well as those roused in patients several times. C-DAK 4000 (Cordis Dow) and CF IS-11 (Baxter Travenol) reused dialyzers obtained from the dialysis clinic were used in the present study. The new dialyzers exhibited a relatively flat APD, whereas saline rinsed and reused dialyzers showed considerable amount of decay. C-DAH dialyzers had a larger APD(11.70